JP5268743B2 - Image communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image communication system which can reduce the deterioration of image quality in image-corrected distortion caused by photographing by a wide angle lens such as an super wide angle lens, a fish-eye lens, etc. <P>SOLUTION: A camera 3 cuts out a target area 13 from a photographed image 11 generated using the super wide angle lens, and transmits coded data of the target area 13. A terminal device 5 decodes the coded data, and corrects the distortion. The distortion correction processing corrects the distortion generated by photographing using the wide angle lens by coordinate conversion from a photographed image space to a correction image space. The wide angle lens such as the super wide angle lens has such a characteristic that the number of coordinate data per a unit area of the image before the correction which is referred by the coordinate conversion is different according to a distance from an imaging center of the photographed image. A coding unit assigns a code amount to each coding block according to a distance from the imaging center to each coding block in the target area. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image communication system including an imaging device including a wide-angle lens and an image receiving device that receives an image from the imaging device.

  2. Description of the Related Art Conventionally, there has been proposed an image communication system that artificially provides a pan / tilt / zoom function using an ultra-wide-angle lens or a fish-eye lens. In this type of system, the camera includes an ultra-wide angle lens or the like, and generates an image with a large angle of view. For example, an image having an angle of view close to 180 ° is generated. However, the captured image is distorted. In accordance with user designation or the like, a part of the region is cut out from the captured image, distortion is corrected, and the corrected region image is displayed.

  In the case of a system that actually includes a pan / tilt mechanism, the pan / tilt mechanism rotates the camera so as to face a desired shooting direction. In the system including the wide-angle lens described above, a region corresponding to the shooting direction is cut out from the captured image. Thereby, although the camera does not actually rotate, it seems to the user that the camera is rotating and photographing a desired direction. Furthermore, a zoom effect can be obtained by enlarging or reducing the image by image processing. Thus, a pseudo pan / tilt zoom function can be provided. Since such pseudo pan / tilt zoom is realized by image processing, it is called software pan / tilt zoom (hereinafter, software PTZ).

  Such a software PTZ system is disclosed in Patent Document 1, for example. This document discloses a camera system using a fisheye lens, which cuts out a part of an imaged region, corrects distortion, and realizes panning, tilting, and zooming of a hemispheric field of view. .

  By the way, in an image communication system, an image is encoded and encoded data is transmitted. In order to reduce the amount of communication data, it is important to reduce the amount of codes.

  For example, the human image compression apparatus described in Patent Document 2 extracts a human image area from an image captured by a camera and sets it as a region of interest. The video encoding unit keeps the image quality high by keeping only the compression rate of the region of interest low, while compressing the image by setting the compression rate of other background image regions high. By such processing, while the central person image has high image quality, the overall compression rate is increased and the total data amount of the image after compression processing is reduced.

  Further, for example, the encoding processing device described in Patent Document 3 includes a face area detection unit, a face verification unit, an ROI determination processing unit, and a video encoding unit. The face area detection unit detects a face area from the video. The face verification unit verifies the validity of the detected face area based on at least one of the position and size of the face in the image. The ROI determination processing unit determines a region of interest (ROI) based on the face region determined to be appropriate by the face verification unit. The video encoding unit encodes the region of interest with a different image quality from other regions, and generates an encoded stream of the moving image.

As described above, conventionally, the attention area in the image is determined as a result of the image recognition, and more codes are assigned to the attention area than the non-attention area. As a result, the amount of code is reduced while maintaining high image quality for a specific region of interest.
Japanese Patent No. 3051173 JP 2001-145101 A JP 2009-5238 A

  However, the conventional image communication system does not consider the characteristics of the projection method of the super-wide-angle lens and the fisheye lens, and does not assign the code amount in the region considering such lens characteristics. Therefore, when the conventional encoding technique is applied to a system using an ultra-wide angle lens or the like, there is a problem that the image quality of the image after distortion correction is deteriorated. The distortion correction here is a process of generating an image that is captured by a normal camera by correcting a large distortion of the super wide-angle lens as described above.

  The above problem will be further described. As projection methods for super wide-angle lenses and fish-eye lenses, for example, orthographic projection and equidistant projection are known. With such a projection method, an image with an extremely large angle of view can be obtained. On the other hand, image distortion that cannot be seen in a normal camera image appears. In the software PTZ described above, such distortion of the captured image is corrected.

  The distortion correction of the super-wide-angle lens is realized by coordinate conversion from a distorted captured image space to a corrected image space from which distortion is removed. However, in the captured image, the subject is greatly compressed in the radial direction as the distance from the imaging center increases. Therefore, in coordinate conversion, the number of coordinate data per unit area before correction referred to is not uniform in the captured image space, and decreases as the distance from the imaging center increases.

  Such a change in the number of referenced data is determined by the projection method of the super-wide-angle lens or fish-eye lens, that is, it can be said to be a characteristic of the super-wide-angle lens or fish-eye lens. Such characteristics are referred to herein as wide-angle lens characteristics.

  Conventionally, a wide-angle lens characteristic is not taken into consideration, and a code amount is uniformly assigned within a region of interest. Therefore, in a portion far from the imaging center, the originally small amount of information is further reduced by encoding. As a result, the image quality of the image after distortion correction is degraded.

  The present invention has been made to solve the conventional problems, and an object of the present invention is to reduce image quality deterioration in an image in which distortion caused by photographing using a wide-angle lens such as a super-wide-angle lens or a fish-eye lens is corrected. It is to provide a communication system.

  The present invention is an image communication system including an imaging device and an image receiving device that receives an image from the imaging device, wherein the imaging device uses a wide-angle lens to generate a captured image; and A region extraction unit that extracts a region of interest from the captured image; and an encoding unit that encodes the region of interest, wherein the image reception device decodes encoded data of the region of interest received from the imaging device And a distortion correction unit that corrects distortion caused by photographing using the wide-angle lens by coordinate conversion from the captured image space to the corrected image space, and the wide-angle lens is referred to in the coordinate conversion. The number of coordinate data per unit area of the pre-correction image is different depending on the distance from the imaging center of the captured image, and the encoding unit includes each of the region of interest from the imaging center. According to the distance to Goka block, it assigns a code amount of the each coding block.

  With this configuration, based on the characteristics of the wide-angle lens, a code amount is assigned to each coding block according to the distance from the imaging center to each coding block in the region of interest. Since a wide-angle lens such as an ultra-wide-angle lens has the above-described characteristics, the number of coordinate data per unit area referred to in the coordinate conversion for distortion correction is reduced in an encoded block far from the imaging center. According to the present invention, a large amount of code is allocated to such an encoded block far from the imaging center. Thereby, it is possible to reduce deterioration in image quality in the image after distortion correction.

  The imaging apparatus of the present invention includes an imaging unit that generates a captured image using a wide-angle lens, a region extraction unit that extracts a region of interest from the captured image, and an encoding unit that encodes the region of interest. Then, the encoding unit controls the code amount of each encoded block so that the code amount is assigned more as the distance from the imaging center of the captured image to each encoded block in the region of interest increases.

  Also with this configuration, the code amount can be suitably controlled based on the characteristics of the wide-angle lens, and the advantages of the present invention can be suitably obtained.

  In the imaging apparatus of the present invention, the area cutout unit is divided into a conversion target area that displays the attention area and a conversion unnecessary area that surrounds the conversion target area and has an outline of a transmission image. The image in the conversion unnecessary area is converted so that the data amount is reduced.

  With this configuration, the amount of codes generated from the conversion unnecessary area can be reduced. Therefore, the overall code amount can be reduced while maintaining the image quality of the conversion target region.

  In the imaging apparatus according to the aspect of the invention, the area cutout unit may paint the conversion unnecessary area with a single color. With this configuration, it is possible to further reduce the amount of code generated from the conversion unnecessary area, and therefore, it is possible to reduce the entire code amount while maintaining the image quality of the conversion target area.

  In the imaging apparatus of the present invention, the encoding unit sets the code amount of the conversion unnecessary area to the minimum amount. With this configuration, it is possible to further reduce the amount of code generated from the conversion unnecessary area, and thus to reduce the overall encoding amount while maintaining the image quality of the conversion target area.

  In the imaging apparatus according to the aspect of the invention, the encoding unit may determine a code amount of an encoding block including a boundary between the conversion target region and the conversion unnecessary region according to a ratio of the conversion target region to the encoding block. Control. With this configuration, the code amount can be adjusted according to the importance of the boundary region, and image quality deterioration during distortion correction can be reduced while suppressing the code amount.

  In the imaging apparatus of the present invention, the area cutout unit converts the size of the attention area into a representative size of the attention area in the captured image space. With this configuration, fluctuations in the communication data amount can be suitably reduced.

  Another aspect of the present invention is an image receiving apparatus, in which code amount allocation of each coding block is performed according to the distance from the imaging center of a captured image obtained using a wide-angle lens to each coding block of a region of interest. A communication unit that receives the encoded data of the region of interest having different values, a decoding unit that decodes the encoded data of the region of interest, and distortion correction that corrects distortion caused by photographing using the wide-angle lens from the decoded data Part. This aspect also provides the above-described advantages of the present invention.

  Another aspect of the present invention is an image processing method, in which a region of interest is cut out from a captured image generated using a wide-angle lens, the extracted region of interest is encoded, and the region of interest from the imaging center of the captured image is encoded. The allocation of the code amount to each coding block is changed according to the distance to each coding block. This aspect also provides the above-described advantages of the present invention.

  The present invention provides a configuration in which a code amount is assigned to each block according to the distance from the imaging center to each block in the region of interest, thereby causing distortion caused by shooting using a wide-angle lens such as a super-wide-angle lens or a fish-eye lens. Therefore, it is possible to provide an image communication system having an effect of reducing the deterioration of image quality in an image corrected for the above.

  Hereinafter, an image communication system according to an embodiment of the present invention will be described with reference to the drawings.

  An image communication system according to an embodiment of the present invention is shown in FIG. First, the outline of the present embodiment will be described with reference to FIG. In FIG. 1, the image communication system 1 has a configuration in which a camera 3 and a terminal device 5 are communicably connected via a network 7. The camera 3 and the terminal device 5 correspond to the imaging device and the image receiving device of the present invention, respectively. The network 7 is an IP network, for example. Since the camera 3 has a function of communicating images via the network 7, it is also referred to as a network camera.

  The camera 3 is provided with a super wide angle lens or a fisheye lens. The super-wide-angle lens or fish-eye lens corresponds to the wide-angle lens of the present invention, and has a very large angle of view (field of view), but is also a lens with a large image distortion due to a large angle of view. In the following description, it is assumed that the camera 3 includes a super wide angle lens. The camera 3 uses a super-wide-angle lens to generate a captured image 11 having a field angle close to 180 ° and corresponding to a hemisphere. The camera 3 cuts out a part of the captured image 11 to generate the attention area 13, encodes the attention area 13, and transmits it to the terminal device 5 via the network 7. In the terminal device 5, the attention area 13 is decoded, distortion is corrected, and a corrected image 15 is obtained. The corrected image 15 is displayed on a monitor, for example.

  The image communication system 1 realizes a software pan / tilt / zoom (software PTZ) function. In a normal mechanical PTZ mechanism, the camera is mechanically rotated in the pan direction and the tilt direction by the rotation mechanism. A zoom function is mechanically realized by the zoom lens. On the other hand, in the configuration of FIG. 1, the attention area 13 is cut out from the captured image 11. Thereby, the same effect as the camera 3 rotating in the direction corresponding to the attention area 13 can be obtained. Further, by enlarging or reducing the attention area 13, the same effect as that of the zoom lens can be obtained. In this way, a pan / tilt zoom function can be realized without a mechanical pan / tilt mechanism and zoom lens. Such a PTZ function is called software PTZ as described above.

  Next, the configuration of the camera 3 and the terminal device 5 will be described with reference to FIGS.

  FIG. 2 shows the configuration of the camera 3. The camera 3 includes an ultra-wide-angle lens 31, an imaging unit 33, an area extraction unit 35, a video encoding unit 37, a network transmission / reception unit 39, an image recognition unit 41, and a lens characteristic holding unit 43.

  The imaging unit 33 includes a solid-state imaging device such as a CCD or a CMOS, and generates an electrical image signal from an optical image obtained by the super wide-angle lens 31. The area cutout unit 35 cuts out the attention area 13 from the captured image 11 generated by the imaging unit 33.

  The attention area 13 may be designated by the user. In this case, data specifying the region of interest 13 is received by the network transmission / reception unit 39 from the terminal device 5 and supplied to the region extraction unit 35. The attention area 13 may be determined based on a detection result by image recognition. In the present embodiment, the image recognition unit 41 detects the motion of the captured image 11 and identifies a moving object in the captured image 11. This motion detection result is supplied to the area cutout unit 35. The area cutout unit 35 determines the attention area 13 so as to include the moving object. Further, the attention area 13 may be determined by combining the user designation and the recognition result.

  The video encoding unit 37 corresponds to the encoding unit of the present invention, and encodes the region of interest 13 cut out by the region cutout unit 35. The video encoding unit 37 includes an encoding control unit 45 and a picture encoding unit 47, and the encoding control unit 45 controls the picture encoding unit 47. The encoding control unit 45 controls the allocation of the code amount for each encoding block in the attention area 13. In the present embodiment, the encoding process is MPEG or the like, and the encoded block is a macroblock. Then, the macroblock coding characteristic is controlled for each macroblock, and thereby the code amount is controlled. The lens characteristic holding unit 43 may be configured by a storage device of the camera 3 and corresponds to the lens characteristic storage unit of the present invention. The lens characteristic holding unit 43 stores wide-angle lens characteristics. The wide-angle lens characteristic is used for code amount control of the video encoding unit 37. Details of the wide-angle lens characteristics and code amount control using the characteristics will be described later.

  The network transmission / reception unit 39 is configured to transmit / receive data to / from the terminal device 5 via the network 7 and corresponds to a communication unit. The network transmission / reception unit 39 receives various instructions from the terminal device 5. The network transmission / reception unit 39 transmits the attention area 13 encoded by the video encoding unit 37 to the terminal device 5.

  FIG. 3 shows the configuration of the terminal device 5. The terminal device 5 includes a network transmission / reception unit 51, a video decoding unit 53, a distortion correction unit 55, a display unit 57, and a user instruction input unit 59.

  The network transmission / reception unit 51 is a communication unit similarly to the network transmission / reception unit 39 of the camera 3, and transmits / receives data to / from the terminal device 5 via the network 7. The network transmission / reception unit 51 transmits a user instruction and other control-related instructions to the camera 3. Further, the network transmission / reception unit 51 receives the encoded data of the attention area 13 from the camera 3.

  The video decoding unit 53 corresponds to the decoding unit of the present invention, and decodes the encoded data of the attention area 13 received by the network transmission / reception unit 51. The distortion correction unit 55 corrects the distortion of the attention area 13 decoded by the video decoding unit 53. The distortion correction is realized by coordinate transformation as will be described later.

  The display unit 57 is a monitor and displays an image after distortion correction. The user instruction input unit 59 is realized by a keyboard, a pointing device, or the like, and inputs a user instruction. In the present embodiment, the position and size of the attention area 13 are designated using the monitor display.

  Next, the image space and the attention area 13 will be described with reference to FIG. As illustrated, two image spaces, a captured image space and a corrected image space, are defined. The captured image space is an image space before distortion correction. Since the angle of view of the super wide-angle lens 31 is close to 180 °, the captured image 11 is indicated by a circle. In the captured image space, there is distortion according to the characteristics of the super wide-angle lens 31. On the other hand, the corrected image space is an image space after distortion correction, and is an image space from which the distortion of the captured image space is removed. The distortion correction is realized by coordinate conversion from the captured image space to the corrected image space.

  As shown in FIG. 4, in the present embodiment, the attention area (ROI area) 13 is divided into a conversion target area 21 and a conversion unnecessary area 23. The conversion target area 21 becomes the display area 25 after distortion correction. The conversion unnecessary area 23 surrounds the conversion target area 21. The outer shape of the conversion unnecessary area 23 corresponds to the outer shape of the image data to be transmitted from the camera 3 to the terminal device 5.

  More specifically, as illustrated, the conversion target area 21 in the captured image space is an area corresponding to the display area 25 in the corrected image space. Here, the display area 25 is rectangular, but the conversion target area 21 is not rectangular because it is before distortion correction. On the other hand, the outer shape of the transmission image data is rectangular. Therefore, there is a gap between the transmission image data and the conversion target area 21. This gap area corresponds to the conversion unnecessary area 23.

  In the present embodiment, the region cutout unit 35 fills the conversion unnecessary region 23 with a single color (pixel value 0). The video encoding unit 37 sets the macro block of the conversion unnecessary area 23 to the skip macro block, and sets the code amount of the conversion unnecessary area 23 to the minimum amount.

  The conversion unnecessary area 23 serves to ensure the shape of the transmission image data. However, the conversion unnecessary area 23 is an unnecessary area on the display side. Therefore, in the present embodiment, by performing the above processing, the data amount of the conversion unnecessary area 23 is reduced, and the entire data amount of the attention area 13 is reduced.

  On the other hand, for the conversion target region 21, the code amount is controlled for each macroblock in the region. This point will be described in detail below.

  5 and 6 show examples of the projection method of the super wide-angle lens 31. FIG. FIG. 5 shows an orthogonal projection method. In this method, each point on the virtual spherical surface is projected perpendicularly to the imaging surface. If the image height is r, the focal length is f, and the angle from the top of the celestial sphere is β, then r = fsinβ. FIG. 6 shows an equidistant projection method. In this method, the image height is proportional to the zenith angle. If the image height is r, the focal length is f, and the zenith angle is θ, then r = fθ. Other projection methods include, for example, a solid angle projection method and a solid angle projection method.

  In the projection method of the super-wide-angle lens 31, a portion away from the imaging center is greatly compressed in the radial direction. Therefore, in the coordinate conversion from the captured image space to the corrected image space, the number of referenced per unit area differs according to the distance from the imaging center. The referred number is the number of coordinate data of the conversion source that is referred to in the coordinate conversion. Here, an encoded block is considered as a unit area. Specifically, the coding block is a macroblock. The number of referenced data per coding block (macroblock) is called “density information”. The density information can be said to be the density of pixels associated with the corrected image.

  FIG. 7 shows the concept of density information. In the captured image space, the attention area 13 includes a conversion target area 21 and a conversion unnecessary area 23. The conversion target area 21 becomes the display area 25 of the corrected image space by coordinate conversion.

  Here, two macroblocks 27 and 29 are shown in FIG. The macro block 27 is farther from the imaging center (the center of the captured image space) than the macro block 29. In this case, the macro block 27 has a smaller number of coordinate data referred to in the coordinate conversion and smaller density information. That is, the density of pixels associated with the corrected image is small.

  The present embodiment focuses on the density information as described above. In the macro block 27, there is little density information. This means that the amount of information referred to in coordinate conversion is small. When such data of the macroblock 27 is further compressed by the encoding process, the amount of information is further reduced and the image quality is deteriorated.

  In order to avoid such a situation, the present embodiment changes the code amount assignment according to the distance from the imaging center to the macroblock. Specifically, the coding macroblock coding characteristics are adjusted. In the present embodiment, code amount allocation is increased by reducing the quantization step. However, within the scope of the present invention, the control of the code amount may not be limited to the above configuration. Parameters other than the quantization step may be controlled.

  FIG. 8A and FIG. 8B show the code amount control described above. FIG. 8A shows density information corresponding to the distance from the imaging center. The density information is A0> A1> A2. Such a change in density information according to the distance from the imaging center is a property based on the projection method of the super-wide-angle lens 31, and is referred to as a wide-angle lens characteristic in the present embodiment. The wide-angle lens characteristic is stored in the lens characteristic holding unit 43 in FIG.

  The video encoding unit 37 refers to the wide-angle lens characteristic of the lens characteristic holding unit 43 and obtains density information of each macroblock in the attention area 13. In this process, density information corresponding to the position of each macroblock in the captured image space is read. Then, the video encoding unit 37 determines the quantization step so that the code amount increases as the density information increases.

  FIG. 8B shows an example of a map for determining the quantization step from the density information. The smaller the density information, the smaller the quantization step and the more the code amount. This map may be stored in the storage device of the camera 3, and the video encoding unit 37 determines a quantization step corresponding to the density information with reference to the map.

  In actual processing, macroblock coding characteristics such as a quantization step are determined by various factors. In this determination process, the quantization step may be adjusted for each macroblock according to the density information. Therefore, the vertical axis in FIG. 8B may be the adjustment amount of the quantization step. The adjustment amount may be a correction amount that is added to or subtracted from the quantization step. The adjustment amount may be a correction coefficient applied to the quantization step by multiplication or division. Other functions may be applied.

  In FIG. 8A and FIG. 8B, the density information is divided into three classes A0 to A3. However, the number of density information classes is not limited to three. More classes may be set. Since the actual density information continuously changes according to the distance from the imaging center, the density information may continuously change in the wide-angle lens characteristics held in the lens characteristic holding unit 43 as in the actual case.

  Further, in FIG. 8A, in order to conceptually explain the lens characteristics, the boundary of the density information is shown as a circle. In practice, the density information is defined in units of blocks. Therefore, in the wide-angle lens characteristic held by the lens characteristic holding unit 43, the boundary of the density information may not be a smooth curve.

  The code amount control for each macroblock in the attention area 13 has been described above. Next, the operation of the image communication system 1 according to the present embodiment will be described.

  In the camera 3, the imaging unit 33 generates the captured image 11 using the super-wide-angle lens 31 and supplies the captured image 11 to the region extraction unit 35 and the image recognition unit 41. The image recognition unit 41 performs a motion detection process on the captured image 11 and supplies the detection result to the region cutout unit 35.

  The area cutout unit 35 cuts out the attention area 13 from the captured image 11. The attention area 13 may be designated by the user. In this case, the user inputs information for specifying the attention area 13 to the user instruction input unit 59. In the present embodiment, a software PTZ function is realized. Therefore, an area to be displayed by the software PTZ may be specified using the user instruction input unit 59. Input information is transmitted from the network transmission / reception unit 51 to the camera 3, received by the network transmission / reception unit 39, and supplied to the encoding control unit 45. In this way, the area cutout unit 35 acquires information for specifying the attention area 13.

  The attention area 13 may be automatically determined based on the detection result by the image recognition by the image recognition unit 41. In this case, the image recognition unit 41 identifies the moving object of the captured image 11. This motion detection result is supplied to the area cutout unit 35. The area cutout unit 35 determines the attention area 13 so as to include the moving object. The above user designation and automatic detection may be combined.

  As shown in FIGS. 4 and 7, the attention area 13 includes a conversion target area 21 and a conversion unnecessary area 23. The area cutout unit 35 fills the conversion unnecessary area 23 in the cut out attention area 13 with a single color.

  The area extracting unit 35 supplies the data of the attention area 13 to the video encoding unit 37. Then, the video encoding unit 37 encodes the attention area 13. In the video encoding unit 37, the encoding control unit 45 controls the picture encoding unit 47.

  FIG. 9 shows the encoding process. The encoding control unit 45 determines whether there is an event (S1), and if there is no event, repeats the determination in step S1. An event is generated in a higher-level control unit of the camera 3, and various components such as the encoding control unit 45 operate according to the event. If there is an event, it is determined whether or not the event is an end event (S3). If it is an end event, the process ends.

  If the event is not an end event, the encoding control unit 45 determines whether the generated event is a video input event (S5). When the attention area 13 is input from the area cutout section 35 to the video encoding section 37, a video input event occurs, and step S5 becomes YES. Then, the video encoding unit 37 generates a macroblock (S7).

  The encoding control unit 45 refers to the information on the wide-angle lens characteristic (FIG. 8A) of the lens characteristic holding unit 43 and acquires the density information of each macroblock (S9). And the encoding control part 45 calculates a macroblock encoding characteristic based on density information (S11). Here, as already described, the macroblock coding characteristics are calculated based on various factors. In this calculation process, density information is referenced. Then, the macroblock coding characteristics are adjusted based on the map of FIG. Specifically, the quantization step is adjusted.

  The encoding control unit 45 supplies the macroblock encoding characteristic calculated in step S11 to the picture encoding unit 47, and causes the picture encoding unit 47 to perform encoding (S13). The picture data obtained in this way is output to the network transmitting / receiving unit 39 (S15).

  FIG. 10 shows details of the picture encoding step S13 of FIG. The process of FIG. 10 is executed by the picture encoding unit 47. In FIG. 10, a macroblock coding characteristic is input from the encoding control unit 45 to the picture encoding unit 47, and a macroblock is input (x01, x02).

  Next, for each macroblock, the picture encoding unit 47 performs a matching process within several frames before and after stored in the frame memory to detect a motion vector, performs motion compensation based on the motion vector information, and performs a prediction image Is generated (x03). In addition, the picture encoding unit 47 performs intra prediction for the macroblock for which motion compensation prediction has been performed, and generates a predicted image (x04). Then, the picture encoding unit 47 performs prediction error calculation for each predicted image at step x03 and step x04, and selects a predicted image having a smaller error (x05).

  Next, the picture encoding unit 47 compares the predicted image selected in step x05 with the input picture and generates a prediction difference signal (x06). Further, the picture coding unit 47 performs DCT transform that decomposes the prediction difference signal into two-dimensional frequency components, and uses the macroblock coding characteristics input in step x01 to convert the DCT transform coefficient into discrete values. The quantization coefficient is output in association with a representative value (x07).

  The macroblock coding characteristic includes a quantization step. As a feature of the present embodiment, the quantization step is adjusted according to the density information of each macroblock. In the macroblock in which the quantization step is adjusted to a small value, the generated code amount increases. On the other hand, the amount of generated code decreases in a macroblock whose quantization step is adjusted to a large value.

  Next, the picture encoding unit 47 performs entropy encoding on the quantized coefficient output at step x07 (x08), and outputs the encoded picture data. The encoding control unit 45 controls the macroblock encoding characteristics so as to satisfy a target reference value (for example, the code amount per second) while receiving feedback of the code amount generated as a result of entropy encoding.

  In addition, the picture encoding unit 47 performs inverse quantization and inverse DCT transform on the quantized coefficient output in step x07 to generate a prediction difference signal (x09). Further, the picture encoding unit 47 adds the prediction image generated by the motion compensation process or the intra prediction process (which is selected based on the determination result of step x05) and the prediction difference signal generated by step x09. Then, a decoded image is generated (x10). Furthermore, the picture encoding unit 47 applies a deblocking filter to the macroblock boundary of the decoded image generated in step x10 so that the block boundary is not conspicuous (x11). Then, the picture encoding unit 47 stores the decoded image subjected to the deblocking process in the frame memory (x12).

  The encoding process by the picture encoding unit 47 has been described above. As described above, the encoding process is performed according to the macroblock encoding characteristics determined by the encoding control unit 45. The macroblock coding characteristics are calculated according to the density information, thereby realizing code amount control according to the density information.

  Further, the conversion unnecessary area 23 of the attention area 13 is filled with a single color before the encoding process. Further, the macroblocks in the conversion unnecessary area 23 are set to skip macroblocks so that the generated code amount is the minimum amount. As a result, the amount of data in the conversion unnecessary area 23 is minimized, and the encoded data is reduced.

  The encoded data of the attention area 13 is supplied from the video encoding unit 37 to the network transmission / reception unit 39 and transmitted from the network transmission / reception unit 39 to the terminal device 5 via the network 7.

  In the terminal device 5, the network transmission / reception unit 51 receives encoded data from the network 7. The encoded data is supplied to the video decoding unit 53 and decoded by the video decoding unit 53. Then, the region of interest 13 after decoding is supplied to the distortion correction unit 55, and distortion correction is performed.

  As described above, in this embodiment, the code amount is controlled according to the density information. Thereby, image quality deterioration in the image after distortion correction can be reduced. The attention area 13 after distortion correction is output to the display unit 57 and displayed.

  The operation of the image communication system 1 according to the embodiment of the present invention has been described above. Next, another embodiment according to the present invention will be described. In this embodiment, the overall configuration and operation of the system are the same as those of the image communication system 1 described above. Furthermore, in this embodiment, the encoding process of the boundary portion between the conversion target area 21 and the conversion unnecessary area 23 in the attention area 13 is improved.

  FIG. 11 shows a part of the attention area 13 and a boundary portion between the conversion target area 21 and the conversion unnecessary area 23. Each square represents a macroblock. The upper four macroblocks include the boundary between the conversion target area 21 and the conversion unnecessary area 23. Here, a macroblock including these boundaries is referred to as a boundary block. In the present embodiment, the code amount is assigned to the boundary block according to the ratio of the conversion target region 21 to the boundary block.

  More specifically, in FIG. 11, “effective pixel ratio” is the ratio of the number of pixels in the conversion target area 21 to the total number of pixels in each macro block 15. The four macroblocks in the lower part of FIG. 11 are completely included in the conversion target area 21. Therefore, as shown in the figure, the effective pixel ratio is 100%. For such macroblocks, a code amount is assigned by the processing described in the above embodiment.

  On the other hand, although not shown, the “effective pixel ratio” is 0% in the macroblock that is entirely included in the conversion unnecessary area 23. Such a macroblock is painted in a single color (0) by the area cutout unit 35 and is set as a skip macroblock.

  Next, the four boundary blocks will be described. As shown in FIG. 11, the boundary line between the conversion target area 21 and the conversion unnecessary area 23 obliquely crosses the four boundary blocks. Therefore, the “effective pixel ratio” is different in the four boundary blocks. The video encoding unit 37 is configured to control the allocation of the code amount to the boundary block according to the effective pixel ratio. Then, the video encoding unit 37 increases the code amount allocation as the effective pixel ratio increases. The encoding control unit 45 may adjust the macroblock encoding characteristics so that the code amount increases as the effective pixel ratio increases.

  FIG. 12 shows the operation of the encoding control unit 45 for performing the above processing. In the process of FIG. 12, step S10 is added to the process of FIG. In step S <b> 10, the encoding control unit 45 calculates the effective pixel ratio of the boundary portion between the conversion target area 21 and the conversion unnecessary area 23. This effective pixel ratio is used in the calculation process of the macroblock coding characteristic in the next step S11. Thereby, the code amount of the boundary block is controlled according to the effective pixel ratio. For example, assume that the effective pixel ratio is 90%. In this case, the macroblock coding characteristics are set so that about 90% of the code amount of the macroblock in the conversion target area 21 is allocated.

  As described above, according to the present embodiment, the allocation of the code amount is controlled according to the size of the ratio occupied by the conversion target area 21 in the macroblock. The code amount of the boundary block can be reduced as compared with the macroblock in the conversion target area 21, and the code amount can be suppressed as a whole. Further, in the boundary block, the code amount increases as the ratio of the conversion target region 21 increases. The size of the ratio of the conversion target area 21 corresponds to the importance of the corresponding macroblock. Therefore, the greater the importance, the greater the amount of codes, and the deterioration of image quality can be reduced.

  In this way, according to the present embodiment, the code amount can be adjusted according to the importance of the boundary region, and image quality deterioration during distortion correction can be reduced while suppressing the code amount.

  Next, another embodiment will be described. Also in this embodiment, the overall configuration and operation of the system are the same as those of the image communication system 1 described above. Furthermore, in this embodiment, the processing of the area extracting unit 35 is improved in consideration of the communication data amount.

  The captured image space has a distortion corresponding to the projection method of the super wide-angle lens 31. Therefore, when the attention area 13 is cut out on the camera side, the size of the attention area 13 changes depending on the cutting position in the captured image space. In this regard, it is assumed that a region having a specific shape (for example, a rectangular region) is moved along the virtual spherical surface of the super-wide-angle lens 31. This specific area corresponds to a corrected image in the corrected image space. In this case, the size of the projected image on the imaging surface changes according to the position from the zenith. This change corresponds to a change in the size of the attention area 13 according to the position in the captured image space as described above.

  When the size of the attention area 13 changes as described above, the amount of encoded data changes, and the transmission amount of the network 7 changes. Therefore, in the present embodiment, the following processing is performed in order to suppress fluctuations in the network transmission amount.

  In the present embodiment, a representative size of the attention area 13 in the captured image space is calculated. The representative size is a representative value of the size of the attention area 13 that changes in the captured image space. In the example of the present embodiment, the representative size is the average size. The average size is, for example, the average of the maximum value and the minimum value of the size of the attention area 13. The average size may be an intermediate value. In this case, the value of the size when the attention area 13 is moved little by little is acquired. Of the large number of size values, the middle value is obtained as the intermediate value.

  The area cutout unit 35 enlarges or reduces the attention area 13 so that the attention area 13 has an average size. Then, the attention area 13 having the average size is encoded by the video encoding unit 37 and transmitted to the terminal device 5. In the terminal device 5, the distortion correction unit 55 enlarges or reduces the average size of the attention area 13 and acquires the original size of the attention area 13. Then, the distortion of the attention area 13 is corrected.

  13 and 14 show an operation according to the present embodiment. FIG. 13 shows the operation of the region cutout unit 35. The area cutout unit 35 determines whether or not there is an event (S21), and if there is no event, repeats the determination in step S21. If there is an event, it is determined whether or not the event is an end event (S23). If it is an end event, the process ends.

  If the event is not an end event, the region extraction unit 35 determines whether the generated event is a region extraction event (S25). If step S25 is YES, the area cutout unit 35 moves the correction image (the image of the correction image space corresponding to the cutout area) on the projection type virtual sphere of the super-wide-angle lens 31 and moves the maximum / maximum cutout area. The minimum size is acquired (S27), and the average size is calculated (S29). In this example, the average of the maximum / minimum sizes is calculated. In another example, an intermediate value of a large number of size values is obtained as described above. Next, the area cutout unit 35 cuts out the area image in the captured image space (S31), enlarges or reduces the cutout area according to the average size (S33), and sets the cutout area of the average size as the attention area 13. The video is output to the video encoding unit 37 (S35).

  Next, FIG. 14 shows the operation of the distortion correction unit of the terminal device 5. The distortion correction unit 55 determines whether there is an event (S41), and if there is no event, repeats the determination in step S41. An event is generated in a higher-level control unit of the terminal device 5, and various configurations of the distortion correction unit 55 operate according to the event. If there is an event, it is determined whether or not the event is an end event (S43). If it is an end event, the process ends.

  If the event is not an end event, the distortion correction unit 55 determines whether the generated event is a distortion correction event (S45). If step S45 is YES, the distortion correction unit 55 reduces or enlarges the decoded image to restore the attention area 13 of the original size (S47). Then, the distortion correction unit 55 performs distortion correction to generate an image of the display region 25 (S49), and outputs the display region 25 after distortion correction to the display unit 57 (S51).

  As described above, according to the present embodiment, the region cutout unit 35 converts the size of the attention region 13 cut out from the captured image 11 into a representative size. Thereby, the fluctuation | variation of network transmission amount can be suppressed.

  The image communication system 1 according to the embodiment of the present invention has been described above. According to the present embodiment, based on the characteristics of the wide-angle lens as described above, a code amount is assigned to each coding block according to the distance from the imaging center to each coding block in the region of interest. Since a wide-angle lens such as an ultra-wide-angle lens has the above-described characteristics, the number of coordinate data per unit area referred to in the coordinate conversion for distortion correction is reduced in an encoded block far from the imaging center. According to the present embodiment, a large amount of code is allocated to such an encoded block far from the imaging center. Thereby, it is possible to reduce deterioration in image quality in the image after distortion correction.

  In addition, according to the present embodiment, the density information shown in FIG. 8A is stored as the wide-angle lens characteristic, and as described with reference to FIG. The amount is controlled. By such processing, it is possible to suitably control the code amount based on the wide-angle lens characteristics, and it is possible to reduce image quality deterioration in the image after distortion correction.

  Further, according to the present embodiment, the attention area 13 is divided into the conversion target area 21 and the conversion unnecessary area 23, and the image of the conversion unnecessary area 23 is converted so that the data amount is reduced by the encoding process. In the above embodiment, the conversion unnecessary area 23 is filled with a single color. The code amount of the conversion unnecessary area 23 is set to the minimum amount. Thereby, the amount of codes generated from the conversion unnecessary area 23 can be reduced. Therefore, the entire code amount can be reduced while maintaining the image quality of the conversion target region 21.

  Moreover, according to this Embodiment, the code amount of the encoding block containing the boundary of the conversion object area | region 21 and the conversion unnecessary area | region 23 is controlled according to the ratio for which the conversion object area | region 21 occupies in an encoding block. As a result, the code amount can be adjusted according to the importance of the boundary region, and image quality deterioration during distortion correction can be reduced while suppressing the code amount.

  Further, according to the present embodiment, the size of the attention area 13 is converted into the representative size of the attention area 13 in the captured image space. Thereby, the fluctuation | variation of communication data amount can be reduced suitably.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and it goes without saying that those skilled in the art can modify the above-described embodiments within the scope of the present invention.

  As described above, the image communication system according to the present invention has an effect of reducing deterioration in image quality in an image in which distortion caused by photographing using a wide-angle lens such as a super-wide-angle lens or a fish-eye lens is corrected. It is useful as an image communication system.

The figure which shows the image communication system in embodiment of this invention Block diagram showing the configuration of the camera of the image communication system The block diagram which shows the structure of the terminal device of an image communication system Diagram showing captured image space and corrected image space The figure which shows the orthographic projection method as an example of the super wide-angle lens projection method Diagram showing equidistant shadow method as an example of super wide-angle lens projection method Diagram showing the concept of density information (A) The figure which shows the wide-angle lens characteristic showing the density information according to the distance from an imaging center (b) The figure which shows the map for determining a macroblock encoding characteristic according to density information Flow chart showing processing of encoding control unit The figure which shows the process of a picture encoding part The figure which shows the boundary part of the conversion object field in the attention field, and the conversion unnecessary field Flow chart showing a modification of the present embodiment Flow chart showing a modification of the present embodiment Flow chart showing a modification of the present embodiment

DESCRIPTION OF SYMBOLS 1 Image communication system 3 Camera 5 Terminal apparatus 7 Network 11 Captured image 13 Attention area 15 Correction | amendment image 21 Conversion object area | region 23 Conversion unnecessary area | region 25 Display area 27, 29 Macroblock 31 Super wide-angle lens 33 Imaging part 35 Area extraction part 37 Image | video Encoding unit 39, 51 Network transmission / reception unit 41 Image recognition unit 43 Lens characteristic holding unit 45 Encoding control unit 47 Picture encoding unit 53 Video decoding unit 55 Distortion correction unit 57 Display unit 59 User instruction input unit

Claims (8)

An imaging unit that generates a captured image using a wide-angle lens;
A region cutout unit that cuts out a region of interest from the captured image;
An encoding unit that encodes the region of interest;
The encoding unit controls the code amount of each encoded block so as to allocate more code amount as the distance from the imaging center of the captured image to each encoded block of the region of interest increases.
The image capturing apparatus according to claim 1, wherein the region cutout unit converts the size of the region of interest into a representative size of the region of interest in a captured image space.   The region cutout unit is divided into a conversion target region for displaying the region of interest and a conversion unnecessary region surrounding the conversion target region and having an outer shape of a transmission image, and the conversion is performed so that the amount of data is reduced by encoding processing. The imaging apparatus according to claim 1, wherein an image of an unnecessary area is converted.   The imaging apparatus according to claim 2, wherein the area cutout unit fills the conversion unnecessary area with a single color.   The imaging apparatus according to claim 2, wherein the encoding unit sets a code amount of the conversion unnecessary area to a minimum amount.   The encoding unit controls a code amount of an encoding block including a boundary between the conversion target region and the conversion unnecessary region according to a ratio of the conversion target region to the encoding block. Item 5. The imaging device according to any one of Items 2 to 4. In order to assign more code amount as the distance from the imaging center of the captured image obtained using the wide-angle lens to the encoded block of the attention area whose size is converted to the representative size in the captured image space increases. A communication unit that receives the encoded data of the region of interest in which the code amount of the encoded block is controlled;
A decoding unit for decoding the encoded data of the region of interest;
An image receiving apparatus comprising: a distortion correction unit that enlarges or reduces the decoded attention area of the representative size, acquires the attention area of the original size, and corrects distortion caused by photographing using the wide-angle lens. A region of interest is cut out from the captured image generated using the wide-angle lens, the size of the extracted region of interest is converted into a representative size in the captured image space,
An image processing method comprising: controlling a code amount of each coding block so that a larger code amount is allocated as a distance from an imaging center of the captured image to each coding block of the region of interest increases. An image communication system comprising an imaging device and an image receiving device that receives an image from the imaging device,
The imaging device
An imaging unit that generates a captured image using a wide-angle lens;
Said to cut out a region of interest from the captured image, the region to convert the size of the target region extracted into typical size of the target area in the captured image space cutting unit,
An encoding unit that encodes the region of interest;
The image receiving device includes:
A decoding unit that decodes the encoded data of the region of interest received from the imaging device;
A distortion correction unit that enlarges or reduces the decoded attention area of the representative size, acquires the attention area of the original size, and corrects distortion caused by photographing using the wide-angle lens;
The wide-angle lens has a characteristic that the number of coordinate data per unit area of the pre-correction image referred to in the coordinate conversion varies depending on the distance from the imaging center of the captured image,
The encoding unit performs encoding as the distance from the imaging center of the captured image obtained using the wide-angle lens to each encoded block of the attention area whose size is converted to a representative size in the captured image space increases. An image communication system, wherein a code amount of each coding block is controlled so as to allocate a large amount.