JP2006197321A - Method and device for processing image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently compress image data. <P>SOLUTION: A one-frame portion is captured from image data, and captured image data are divided into high quality image data for displaying an image with a high image quality and low quality image data for displaying with a degraded quality, in accordance with a prescribed area. When captured image data have a prescribed frame, low quality image data are stored, and the low quality image data are read out, and high quality image data and the read low quality image data are synthesized, and synthesized image is compressed, whereby image data can be efficiently compressed. This invention is applicable to image processing methods. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing method, apparatus, and program, and more particularly, to an image processing method, apparatus, and program capable of efficiently compressing image data.

  In recent years, moving images are compressed and transmitted via a network, such as a videophone, and moving images are compressed and recorded on a recording medium, such as an HDD (Hard Disk Drive) recorder or DVD (Digital Versatile Disk) recorder. The equipment that can be increased. In addition, it is expected that an environment in which AV (Audio Visual) devices connected to a network compress a moving image and transmit the compressed image via the network in a general home is expected.

  Conventionally, when transmitting a moving image, the image data is generally compressed after being transmitted due to the limitation of the bandwidth of the transmission path. Similarly, when a moving image is recorded on a recording medium, the image data is generally recorded after being compressed due to the limitation of the capacity of the recording medium. Furthermore, if the input signal is an analog signal, A / D conversion (Analog / Digital Conversion) is performed to convert the analog signal to a digital signal, and then a codec such as MPEG (Moving Picture Experts Group). Data is compressed by encoding (encoding) using (CODEC (Compression / Decompression)).

  On the other hand, the side receiving the encoded image data decodes the encoded image data to obtain the original image data. As described above, in the case of image data that is digital data, the image data is compressed by encoding, so that the predetermined processing can be executed after the capacity of the image data is reduced.

  In addition, there is a moving image encoding apparatus that can set a priority in advance and preferentially encode a portion set to a high priority such as a focused portion of an image, thereby encoding the portion with high image quality ( For example, see Patent Document 1).

Japanese Patent Laid-Open No. 5-95540

  However, when the encoded image data is decoded, there is a problem that if the compression method of the codec is changed, the image data cannot be decoded unless it is a decoder suitable for the method. For example, the moving picture coding apparatus disclosed in Japanese Patent Laid-Open No. 5-95540 has to change the decoder on the playback side in accordance with the compression method of the codec used for encoding.

  In addition, when the codec compression method is changed to a high compression method, the load on the compression process increases, so that the encoder needs a certain degree of processing capability.

  Furthermore, a desired compression rate may not be ensured due to limitations on the bandwidth of the transmission path and the capacity of the recording medium. In this case, although the compression rate has to be reduced, there is a problem that the quality of the entire image deteriorates when decoded. In addition, even when it is desired to improve the image quality, there has been a problem that the compression rate cannot be increased due to limitations on the bandwidth of the transmission path and the capacity of the recording medium.

  The present invention has been made in view of such a situation. Before encoding, image data is divided into image data in a region where high image quality is to be obtained and image data in a region where high image quality is not high, so that the image data is efficiently processed. So that it can be compressed.

  According to the image processing method of the present invention, each of the first image data for displaying each of the pictures for displaying the moving image is converted into the first data and the second data according to a predetermined area on the picture. A division step of dividing the first image data of a predetermined number of pictures displayed in a predetermined period, a storage step of storing second data divided from the first image data of the predetermined picture, and a period Each of the predetermined number of pictures of the period from each of the first data divided from each of the first image data of the predetermined number of pictures and the recorded second data. Generating the second image data for displaying the image, and encoding for encoding the second image data by the encoding method for compression encoding based on the difference between the pictures Characterized in that it comprises a step.

  The image processing method of the present invention further includes a correction step of correcting the second data so as to replace the value of the lower-order bit of the predetermined number of digits of the pixel value of the second data with a predetermined value. It is characterized by including.

  Furthermore, the image processing method of the present invention includes a detection step of detecting a movement amount that is a moving amount of an image of a subject in a picture, and the first image data and the second data based on the movement amount. And a determination step for determining an area to be divided into data.

  The image processing method of the present invention also includes a measurement step for measuring a delay time in a network for transmitting image data to the other party, and the first image data and the second data based on the delay time. And a determination step for determining an area to be divided into data.

  The image processing apparatus according to the present invention uses first data, second data, and first image data for displaying each of the pictures for displaying a moving image according to a predetermined area on the picture. A dividing unit that divides the first data of the predetermined number of pictures displayed in a predetermined period, a storage unit that stores the second data divided from the first image data of the predetermined picture, and a period Each of the predetermined number of pictures of the period from each of the first data divided from each of the first image data of the predetermined number of pictures and the recorded second data. Generating means for generating each of the second image data for displaying the image, and encoding means for encoding the second image data by an encoding method for compression encoding based on a difference between pictures. It is characterized in.

  The program of the present invention divides each first image data for displaying each picture for displaying a moving image into first data and second data according to a predetermined area on the picture. A dividing step, a storing step for storing second data divided from the first image data of a predetermined picture among a predetermined number of pictures displayed in a predetermined period, and a period of Each of the predetermined number of pictures of the period is displayed from each of the first data divided from each of the first image data of the predetermined number of pictures and the recorded second data. A generating step for generating each of the second image data to be encoded, and an encoding step for encoding the second image data by an encoding method for compression encoding based on a difference between pictures. Characterized in that it comprises a-up.

  In the image processing method, apparatus, and program of the present invention, each of the first image data for displaying each of the pictures for displaying the moving image is changed to the first data according to a predetermined area on the picture. And second data divided from the first image data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period, and stored in a period Each of the predetermined number of pictures of the period from each of the first data divided from each of the first image data of the predetermined number of pictures and the recorded second data. Each of the second image data to be displayed is generated, and the second image data is encoded by an encoding method in which compression encoding is performed based on the difference between pictures.

  According to the present invention, image data can be efficiently compressed.

  In addition, according to the present invention, image data for efficient compression is supplied to the encoder, so that the existing encoder and decoder can be used as they are.

  Furthermore, according to the present invention, it is possible to reduce the amount of data while maintaining the image quality of an image area required to have high image quality.

  BEST MODE FOR CARRYING OUT THE INVENTION The best mode of the present invention will be described below. The correspondence relationship between the disclosed invention and the embodiments is exemplified as follows. Although there are embodiments which are described in this specification but are not described here as corresponding to the invention, the embodiments correspond to the invention. It does not mean that it is not a thing. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. It is not a thing.

  Further, this description does not mean all the inventions described in the specification. In other words, this description is for the invention described in the specification and not claimed in this application, i.e., for the invention that will be applied for in the future or that will appear as a result of amendment and added. It does not deny existence.

  According to the present invention, an image processing method is provided. In this image processing method, each of first image data for displaying each picture for displaying a moving image is divided into first data and second data according to a predetermined area on the picture. A dividing step (for example, the process of step S13 in FIG. 4), and a second divided from the first image data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period A storage step of storing data (for example, the process of step S15 in FIG. 4), and each of the first data divided from each of the first image data of a predetermined number of pictures in the period Generating step (for example, the step of FIG. 4) for generating each of the second image data for displaying each of the predetermined number of pictures of the period from the second data. It includes the process of S16), based on the difference between pictures, the coding method for compressing and encoding, and an encoding step of encoding the second image data (for example, step S17 in FIG. 4).

  In addition, this image processing method corrects the second data so as to replace the value of the lower-order bit of the predetermined number of digits of the pixel value of the second data with a predetermined value (for example, FIG. 8 (step S36).

  Furthermore, this image processing method uses the detection step (for example, the process of step S53 in FIG. 10) for detecting the amount of motion that is the amount of movement of the subject image in the picture, and the first image data based on the amount of motion. And a determination step (for example, the process of step S54 in FIG. 10) for determining an area for dividing the data into the first data and the second data.

  In addition, this image processing method uses the measurement step for measuring the delay time in the network for transmitting the image data to the other party (for example, the process of step S105 in FIG. 15) and the first image data based on the delay time. Further includes a determination step (for example, the process of step S74 of FIG. 14) for determining an area for dividing the data into the first data and the second data.

  According to the present invention, an image processing apparatus is provided. The image processing apparatus divides each first image data for displaying each picture for displaying a moving image into first data and second data according to a predetermined area on the picture. Dividing means (for example, the image dividing unit 51 in FIG. 3), and a second divided from the first image data of a predetermined picture among a predetermined number of pictures displayed in a predetermined period Storage means for storing data (for example, the image storage unit 52 in FIG. 3), and each of the first data divided from each of the first image data of a predetermined number of pictures in the period Generating means (for example, the image composition unit 53 in FIG. 3) for generating each of the second image data for displaying each of the predetermined number of pictures in the period from the second data, and between the pictures Based on the difference, the coding method for compressing and encoding comprises encoding means for encoding the second image data (e.g., the encoder 15 of FIG. 2) and.

  According to the present invention, a program is provided. In this image processing method, each of the first image data for displaying each of the pictures for displaying the moving image is converted into the first data and the second data according to a predetermined area on the picture. Dividing into data (for example, the process of step S13 in FIG. 4) and the first picture data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period. Storage step for storing the second data (for example, the process of step S15 in FIG. 4), and each of the first data divided from each of the first image data of a predetermined number of pictures in the period And a second generation step of generating each of the second image data for displaying each of a predetermined number of pictures in a period from the recorded second data (for example, For example, the process of step S16 in FIG. 4 and the encoding step for encoding the second image data by the encoding method for compression encoding based on the difference between pictures (for example, in step S17 in FIG. 4). Process).

  This program can be stored in a storage medium (for example, the magnetic disk 321 in FIG. 16).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an embodiment of a videophone system to which the present invention is applied.

  The videophone device 1-1 is, for example, a device dedicated to a videophone, a personal computer connected to a camera, an IP (Internet Protocol) videophone (IP phone or TV compatible with a videophone), or a mobile phone with a built-in camera. Composed.

  The videophone device 1-1 transmits image data or audio data to the videophone device 1-2 via the network 2, or receives image data or audio data from the videophone device 1-2. For example, the videophone device 1-1 transmits image data of a face image of a user who uses the videophone device 1-1 or voice data of a talking voice via the network 2 to the videophone device 1-2. Or the image data of the face image of the other party using the videophone device 1-2 or the voice data of the talking voice is received.

  Similarly to the video phone device 1-1, the video phone device 1-2 is composed of a device dedicated to the video phone, etc., and transmits image data or audio data to the video phone device 1-1 via the network 2. Alternatively, image data or audio data is received from the videophone device 1-1.

  The network 2 includes, for example, a mobile network, a PSTN (Public Switched Telephone Network) such as ISDN (Integrated Services Digital Network), the Internet, and the like, according to a predetermined protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol). The videophone device 1-1 and the videophone device 1-2 can access each other.

  Hereinafter, when it is not necessary to distinguish the videophone device 1-1 and the videophone device 1-2 from each other, they are simply referred to as the videophone device 1.

  FIG. 2 is a block diagram showing a configuration of an embodiment of a videophone device 1 to which the present invention is applied. The videophone device 1 is an example of an image processing device of the present invention.

  The camera 11 is composed of a CCD (Charge Coupled Device) sensor or the like. The camera 11 captures light from a subject (not shown) and supplies image data corresponding to the image of the subject to an A / D converter 13 via an input terminal 12 which is a so-called connection interface. To do. Note that the image sensor that constitutes the camera 11 is not limited to a CCD sensor, and may be any device that generates an image signal in units of pixels, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor.

  The A / D converter 13 converts the image data that is an analog signal supplied from the input terminal 12 into image data that is a digital signal. The A / D conversion unit 13 supplies image data that is a digital signal generated by the conversion to the image processing unit 14.

  The image processing unit 14 controls the image data supplied from the A / D conversion unit 13 based on the control of a CPU (Central Processing Unit) 16, for example, the image data in a region for high image quality and the high image quality. A predetermined process such as a process of dividing the image data in the area not to be processed is executed, and the image data subjected to the predetermined process is supplied to the encoder 15. Details of the image processing unit 14 will be described later.

  Under the control of the CPU 16, the encoder 15 converts the image data supplied from the image processing unit 14 according to a scheme such as MPEG2, MPEG4, H.263, or H.264 (MPEG4 AVC (Advanced Video Coding)). Encode. The encoder 15 supplies the encoded image data to the CPU 16.

  The encoding method is not limited to the above-described example, and any encoding method may be used as long as the encoding is performed using information on the difference between a certain still image and other still images. For example, any method may be used as long as encoding is performed using difference information between a certain frame and the previous frame.

  Although not shown in the figure, in the videophone 1, the audio data captured from the microphone (microphone) is also encoded by the audio encoder and processed in the same manner as the image data.

  The CPU 16 controls each part of the videophone device 1. The CPU 16 acquires the encoded image data supplied from the encoder 15 and supplies the acquired image data to the communication unit 19.

  Further, the CPU 16 reads and executes a program recorded in the ROM 17 as necessary. The CPU 16 supplies data to the RAM 18 as necessary, or acquires data temporarily stored in the RAM 18.

  The communication unit 19 transmits the image data supplied from the CPU 16 to another videophone device (for example, the videophone device 1-2) via the network 2.

  FIG. 3 is a block diagram illustrating a detailed configuration of the image processing unit 14.

  The image processing unit 14 sequentially acquires the image data supplied from the A / D conversion unit 13 under the control of the CPU 16 and captures one frame of the acquired image data. For example, the image processing unit 14 performs predetermined processing such as processing for dividing the captured image data into image data in an area where high image quality is desired and image data in an area where high image quality is not high. The processed image data is supplied to the encoder 15.

  The image processing unit 14 is configured to include an image dividing unit 51, an image storage unit 52, and an image composition unit 53.

  The image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. The image dividing unit 51 divides the captured image data according to a predetermined area of the image under the control of the CPU 16. The image dividing unit 51 supplies the divided image data to the image storage unit 52 or the image composition unit 53.

  For example, the image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13 under the control of the CPU 16 and makes the captured image data a high quality area (hereinafter, high quality area). Image data (hereinafter referred to as “high quality image data”) and image data (hereinafter referred to as “low quality image data”) corresponding to an area that does not have high image quality (hereinafter referred to as “low quality image area”). Of the divided image data, high-quality image data is supplied to the image composition unit 53, and low-quality image data is supplied to the image storage unit 52.

  The image storage unit 52 stores the image data supplied from the image dividing unit 51. The image storage unit 52 supplies the stored image data to the image composition unit 53 in response to a request from the image composition unit 53.

  For example, the image storage unit 52 stores the low-quality image data supplied from the image dividing unit 51, and stores the stored low-quality image data in the image composition unit 53 in response to a request from the image composition unit 53. Supply.

  The image composition unit 53 reads out image data from the image storage unit 52, and combines the image data based on the read image data and the image data supplied from the image division unit 51. The image synthesis unit 53 supplies the synthesized image data to the encoder 15.

  For example, the image composition unit 53 reads out low-quality image data from the image storage unit 52 and synthesizes the image data based on the read-out low-quality image data and high-quality image data supplied from the image division unit 51. Then, the synthesized image data is supplied to the encoder 15.

  In other words, the image composition unit 53 reads the low-quality image data from the image storage unit 52, and based on the read low-quality image data and the high-quality image data supplied from the image dividing unit 51, the image data It can be said that the image data is generated by combining the images. The same applies to the following description.

  By the way, in the videophone system of FIG. 1, for example, when a conversation is made by videophone between the videophone device 1-1 used by a certain user and the videophone device 1-2 used by the other party, The communication unit 19 of the videophone device 1-1 transmits the image data of the user captured by the camera 11 to the videophone device 1-2 used by the other party, so that the other party sends the videophone device 1-2 to the videophone device 1-2. It becomes possible to have a conversation while looking at the displayed image of the user. In addition, by performing the same process in the videophone device 1-2 used by the other party, the user and the other party can talk while watching the video in both directions. Hereinafter, the videophone device 1 that executes such processing will be described.

  With reference to the flowchart of FIG. 4, image compression processing by the videophone device 1 will be described.

  In step S <b> 11, the camera 11 captures a subject and supplies image data that is an analog signal of the captured subject to the A / D conversion unit 13 via the input terminal 12. For example, the camera 11 captures an image near the face of the user who is using the videophone device 1, and receives image data that is an analog signal of the captured image near the user face via the input terminal 12. The data is supplied to the D converter 13.

  In step S <b> 12, the A / D conversion unit 13 converts image data that is an analog signal supplied from the camera 11 via the input terminal 12 into image data that is a digital signal. The A / D conversion unit 13 supplies image data that is a digital signal generated by the conversion to the image processing unit 14. For example, in step S12, the A / D converter 13 converts the image data of the image near the user's face, which is an analog signal supplied from the camera 11 via the input terminal 12, into image data that is a digital signal. The converted image data is supplied to the image processing unit 14.

  In step S <b> 13, the image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13, captures one frame of the acquired image data, and captures the captured image data as the CPU 16. Based on this control, the image data is divided into high-quality image data and low-quality image data.

  Note that the process of capturing image data for one frame from the image data may be executed by, for example, the A / D converter 13 without being executed by the image dividing unit 51.

  By the way, in general, a moving image expresses a motion by quickly switching one by one of still images (frames). That is, the moving image is displayed by a predetermined number of frames capable of expressing the movement in a predetermined period. In this case, the sound is synchronized with the moving image. For example, the image dividing unit 51 uses this frame to divide the image data into high-quality image data and low-quality image data. Also, for example, a single still image may be handled as a picture (for example, an I picture, a B picture, a P picture, etc.), such as GOP (Group Of Pictures) in which several pieces of image data are grouped. However, here, a frame will be described as an example of a picture.

  Hereinafter, with reference to FIG. 5 and FIG. 6, a process of dividing the image data in each frame by the image dividing unit 51 into high-quality image data and low-quality image data will be described.

  First, the details of the frame will be described with reference to FIG.

  FIG. 5 is a diagram showing a frame displayed at a certain time. Further, the horizontal direction in the figure represents the time axis, and the time will be described as passing from the left to the right in the figure as indicated by the arrow on the lower side in the figure.

  Here, when the frame 1-1 is displayed at the time ta, the frames 1-2 to 1-30 are displayed in one second until the time tb. That is, the motion as a moving image is expressed by continuously displaying the frames 1-1 to 1-30 (30 frames) for one second from the time ta to the time tb. Similarly, by displaying frames 2-1 to 2-30 (30 frames) continuously for 1 second from time tb to time tc, motion as a moving image is expressed. The same applies to frames 3-1 to 3-30 (not shown), frames 4-1 (not shown) to 4-30 (not shown),.

  Also, how many frames are displayed per second can be determined according to the characteristics of the device used and the application program. For example, by increasing the frame rate (the number of frames displayed per second), the image quality of the displayed image can be made smoother (higher image quality), while the frame rate is lowered. If this happens, the image quality will not be smooth and frames may be dropped (not high image quality). Specifically, in the example shown in FIG. 5, the frame rate is 30 (FPS (Frame Per Second)). That is, the frames 1-1 to 4-30 are displayed every 1/30 seconds.

  Furthermore, the image quality of the displayed image also changes depending on how much data is flowed per second. For example, by increasing the bit rate (the amount of data that flows per second), the image quality of the displayed image can be improved, whereas when the bit rate is decreased, the image quality deteriorates ( (There is no possibility of high image quality). Also, there is a trade-off between the bit rate and the image quality of the image to be displayed. When the bit rate is increased, the data amount (file size) increases, and when the bit rate is decreased, the data amount (file size) decreases. .

  Further, even in one frame, there are an area where the image quality is high because it is an important part (a part to be noticed), and an area where the image quality does not need to be improved because it is not so important.

  FIG. 6 is a diagram illustrating an example of a screen of a display unit (not shown) that displays an image captured by the camera 11.

  The screen 101 defines 480 (dots) × 640 (dots) (so-called VGA (Video Graphics Array)) as the resolution of the display area.

  For example, the image displayed on the screen 101 is an image near the face of the user who is using the videophone device 1-1 captured by the camera 11, and the user and the videophone device 1-2 have a conversation. It is also an image that the other party is watching.

  The A area 111 is an image of a low quality area that does not need to improve the image quality among the images displayed on the screen 101, and the B area 112 is a high image of the images displayed on the screen 101. It is an image in a high-quality area to achieve image quality. For example, in the case of the videophone device 1, the important area (the area that is noticed by the other party) is an image near the center in which the user's face and body are reflected, and the other background areas are almost the same for the other party. Often images are not important without attention.

  Therefore, in the example shown in FIG. 6, 10% of the periphery of the image displayed on the screen 101 is set as the A area 111 as an image of the low quality area such as the background, and the other face and body of the user are reflected. An image near the center is set as a B area 112 which is a high image quality area. That is, an area having a resolution of vertical 384 (dots) × horizontal 512 (dots) excluding 10% around the image displayed on the screen 101 is the B area 112.

  In this case, 10% around the image means a 10% area around the resolution of the display area on the screen 101 (for example, vertical 480 (dots) × horizontal 640 (dots)). For example, in the example shown in FIG. 6, the outer region is surrounded by vertical 48 (dots) and horizontal 64 (dots). Hereinafter, the description will be made in the same manner.

  In addition, the method for specifying the areas in the A area 111 and the B area 112 may be a fixed area specified in advance by user settings or factory default settings, or as described later, It may be a variable area that dynamically changes depending on the state of the network 2. Furthermore, the method of designating the area is not limited to the ratio of the A area 111 or the B area 112 to the entire image, but may be designated by, for example, pixels (dots).

  Further, the method of specifying the area is not limited to the method of specifying the rectangular (square) area shown in FIG. 6, and any method can be used as long as the area on the screen 101 can be specified, such as a circle, an ellipse, or a triangle. Good. The area designation method is not limited to the area designation method with the center of the screen 101 shown in FIG. 6 as the base point. For example, the area on the screen 101 is designated based on the right corner, the left corner, or the like. May be.

  Hereinafter, information indicating the areas of the high image quality area (B area 112) and the low image quality area (A area 111) will be referred to as area information.

  Returning to the flowchart of FIG. 4, for example, in step S <b> 13, the image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13, and the captured image data for one frame is controlled by the CPU 16. Is divided into image data corresponding to the A area 111 and image data corresponding to the B area 112 in FIG.

  That is, the CPU 16 is, for example, data set by the user, and is an area indicating that “10% around the image is the A area 111 and the other image is the B area 112” recorded in the ROM 16. Based on the information, the image division unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13, and the captured image data for one frame is converted into image data corresponding to the A area 111 in FIG. And image data corresponding to the B area 112. Note that the processing for designating the area in the processing of step S13 may be performed by the image processing unit 14 instead of the CPU 16.

  In step S <b> 14, the image dividing unit 51 determines whether or not the frame corresponding to the image data that has been divided is a predetermined frame. For example, in step S14, the image dividing unit 51 displays the frame corresponding to the divided image data at the head of the frame displayed for 1 second (for example, the frame 1-1, the frame 2-1, the frame of FIG. 3-1...

  Further, for example, in step S14, the image dividing unit 51 determines whether or not the frame corresponding to the divided image data is the head of a GOP in which several pieces of image data are collected. Further, for example, in step S14, the image dividing unit 51 determines whether or not the frame corresponding to the divided image data is a frame encoded as an I picture in the GOP. In this case, the image dividing unit 51 acquires data indicating that each frame is encoded as one of an I picture, a B picture, and a P picture from the encoder 15 via the CPU 16.

  If it is determined in step S14 that the frame is a predetermined frame, the process proceeds to step S15, and the image dividing unit 51 supplies the low-quality image data among the divided image data to the image storage unit 52, and the high-quality image data. Is supplied to the image composition unit 53. The image storage unit 52 stores the low image quality image data supplied from the image dividing unit 51.

  For example, in step S <b> 15, the image dividing unit 51 supplies image data corresponding to the A region 111 to the image storage unit 52 among the divided image data, and the image data corresponding to the B region 112 to the image composition unit 53. Supply. The image storage unit 52 stores image data corresponding to the A area 111 supplied from the image dividing unit 51. That is, the image data corresponding to the A area 111 stored in the image storage unit 52 corresponds to the A area 111 of the first frame (for example, the frame 1-1 in FIG. 5) among the frames displayed for one second. Only the image data to be stored is stored.

  On the other hand, if it is determined in step S14 that the frame is not a predetermined frame, the process of step S15 is skipped and the process proceeds to step S16. The image composition unit 53 reads the low-quality image data from the image storage unit 52 and reads it. Based on the low-quality image data and the high-quality image data supplied from the image dividing unit 51, the image data is synthesized. The image synthesis unit 53 supplies the synthesized image data to the encoder 15.

  For example, in step S <b> 16, the image composition unit 53 reads image data corresponding to the A region 111 in the first frame from the image storage unit 52, and is supplied from the image data corresponding to the read A region 111 and the image dividing unit 51. Based on the image data corresponding to the B area 112, the image data corresponding to the A area 111 and the image data corresponding to the B area 112 are combined, and the combined image data is supplied to the encoder 15.

  In this way, for example, in the frame 1-1, the image data corresponding to the A area 111 becomes image data corresponding to the A area 111 of the frame 1-1, and the image data corresponding to the B area 112 is The image data corresponds to the B area 112 of the frame 1-1. In the frame 1-2, the image data corresponding to the A area 111 is image data corresponding to the A area 111 of the frame 1-1, and the image data corresponding to the B area 112 is the B area of the frame 1-2. 112 corresponds to the image data. Further, in the frame 1-3, the image data corresponding to the A area 111 is image data corresponding to the A area 111 of the frame 1-1, and the image data corresponding to the B area 112 is the B area of the frame 1-3. 112 corresponds to the image data. Similarly, in the frames 1-4 to 1-30, the image data corresponding to the A area 111 is image data corresponding to the A area 111 of the frame 1-1, and the B area 112 is stored in the frames 1-4 to 1-1. The image data corresponds to each B region 112 of −30.

  Similarly, in the frames 2-1 to 2-30, the image data corresponding to the A area 111 is image data corresponding to the A area 111 of the frame 2-1, and the image data corresponding to the B area 112 is , Image data corresponding to each B area 112 of the frames 2-1 to 2-30. The same applies to frames 3-1 to 3-30 (not shown), frames 4-1 (not shown) to 4-30 (not shown),.

  That is, the image composition unit 53 reads the image data corresponding to the A area 111 (image data corresponding to the A area 111 of the first frame) stored in the image storage section 52, and the image corresponding to the B area 112 is read. By combining with the data, the image data corresponding to the A area 111 of the frame displayed per second does not change, and only the image data corresponding to the B area 112 changes.

  In step S <b> 17, the encoder 15 encodes the image data supplied from the image composition unit 53 according to, for example, the MPEG4 method, and supplies the encoded image data to the CPU 16.

  That is, the image data encoded by the encoder 15 is such that the same image data is used for the image data of the low image quality area that does not have high image quality for each frame displayed per second by the image processing unit 14. Therefore, the difference data when compressing the frame can be reduced for the low image quality area, so the data amount of the low image quality area can be reduced and the data amount of the high image quality area for high image quality can be increased. As a result, the image quality of an important region can be improved.

  For example, when the bit rate is constant, the image data corresponding to the A area 111 of the frame 1-1 is used for each A area 111 in the frames 1-1 to 1-30 displayed per second. Thus, since the image data corresponding to the A area 111 of the frames 1-2 to 1-30 is not used, the A area 111 is included in the difference data when the frames 1-2 to 1-30 are compressed. Therefore, the image data corresponding to the A area 111 can be reduced, and the image data corresponding to the B area 112 can be increased to improve the image quality of the B area 112. Can be made.

  More specifically, in the example shown in FIG. 6, 10% of the periphery of the image displayed on the screen 101 is set as the A area 111, and other images near the center where the user's face and body are reflected are set. Since the B area 112 is set, the ratio of the B area 112 in the image displayed on the screen 101, that is, the resolution of the image displayed on the screen 101: the resolution of the B area 112 is 480 (dots) × Horizontal 640 (dots): Vertical 384 (dots) × horizontal 512 (dots), so the ratio is 64%. Similarly, the proportion of the area A 111 in the image displayed on the screen 101 is 36%.

  That is, in the frames 1-2 to 1-30, the image data corresponding to the A area 111 occupying 36% of the image displayed on the screen 101 is not used. The image quality of the B area 112 can be improved by increasing the amount of image data corresponding to the B area 112 that occupies%.

  In the example shown in FIG. 6, the data amount of the image data of the frames 1-1 to 1-30 is compared with the case where the present invention is not applied, that is, the amount of data when the present invention is not applied. The ratio to the amount of data when the invention is applied is 480 (dots) x 640 (dots) x 30 (frames): 480 (dots) x 640 (dots) x 1 (frame) + 384 (vertical) dots) × 512 (dots) × 29 (frames), so the ratio is 65.2%. That is, by applying the present invention, for example, the maximum data amount can be reduced by 34.8%.

  In step S <b> 18, the CPU 16 acquires the encoded image data supplied from the encoder 15 and supplies the acquired image data to the communication unit 19. The communication unit 19 transmits the image data supplied from the CPU 16 to the videophone device 1-2 used by the other party via the network 2, returns to the process of step S11, and repeats the above process.

  As described above, one frame is divided into a high-quality area and a low-quality area, and the same image data is used for the low-quality image data corresponding to the low-quality area among the frames displayed in one second. By doing so, the image quality of the high-quality area can be improved when encoding the image data. That is, according to the present invention, the image quality can be maintained even when the moving image compression rate is lowered, and the image quality can be improved without increasing the moving image compression rate.

  In the present invention, since the image processing unit 14 performs a process of dividing the image data into a high image quality region and a low image quality region before the encoder 15 encodes the image data, the videophone device 1 that encodes the image data. -1 encoder 15 and the decoder (not shown) of the videophone apparatus 1-2 that decodes the encoded image data need not be changed at all, and existing encoders and decoders are used as they are. can do. That is, there is an advantage that the function of the present invention can be easily incorporated (commercialized easily) into an existing encoder or decoder. Further, the low image quality region can be reproduced with a sufficient image quality although there is some degradation in image quality.

  By the way, in the above-described A region 111 such as the background, the image displayed on the screen looks like a still image, but in reality, it constantly changes due to errors and noise due to quantization. ing. Therefore, in the present invention, it is possible to improve the image quality of the B region 112 by suppressing the difference data generated by the error and noise due to the quantization and reducing the data amount of the A region 111.

  Next, processing for suppressing difference data of image data corresponding to a low image quality region will be described with reference to FIGS. In the case of executing the process of suppressing the difference data of the image data corresponding to the low image quality area, the image processing unit 151 is provided instead of the image processing unit 14 in FIG.

  FIG. 7 is a block diagram illustrating a detailed configuration of the image processing unit 151. The same parts as those shown in FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image processing unit 151 is provided instead of the image processing unit 14 of FIG. The image processing unit 151 sequentially acquires the image data supplied from the A / D conversion unit 13 under the control of the CPU 16 and captures one frame of the acquired image data. The image processing unit 151 executes predetermined processing such as processing for dividing the captured image data into image data in an area where high image quality is to be achieved and image data in an area where high image quality is not high, for example. The processed image data is supplied to the encoder 15.

  The image processing unit 151 is configured to include an image dividing unit 51, an image storage unit 52, an image composition unit 53, and a correction unit 161.

  The image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. The image dividing unit 51 divides the captured image data for one frame into high-quality image data and low-quality image data according to a predetermined area of the image under the control of the CPU 16. Of the divided image data, the image dividing unit 51 supplies high-quality image data to the correction unit 161 and supplies low-quality image data to the image storage unit 52.

  The image storage unit 52 stores the low image quality image data supplied from the image dividing unit 51. The image storage unit 52 supplies low-quality image data to the correction unit 161 in response to a request from the correction unit 161.

  The correction unit 161 reads the low-quality image data from the image storage unit 52 and corrects each image data based on the read low-quality image data and the high-quality image data supplied from the image dividing unit 51. The correction unit 161 supplies the corrected high-quality image data and low-quality image data to the image composition unit 53.

  For example, the correction unit 161 reads the low-quality image data from the image storage unit 52, and the screen corresponding to the low-quality image data based on the read low-quality image data and the high-quality image data supplied from the image division unit 51. Among the bit lengths of the pixels of the image displayed on the screen, the predetermined lower bits are corrected so as to have a fixed value, and the corrected high-quality image data and low-quality image data are supplied to the image composition unit 53.

  The image synthesis unit 53 synthesizes the corrected high-quality image data and low-quality image data supplied from the correction unit 161. The image synthesis unit 53 supplies the synthesized image data to the encoder 15.

  Note that by removing the image storage unit 52 from the configuration, the correction unit 161 corrects each of the high-quality image data and the low-quality image data supplied from the image division unit 51, and the corrected image data is converted into the image composition unit 53. You may make it supply to. That is, in this case, the low-quality image data in the frame displayed for one second is not the same image data, but is all different image data, but the image data is encoded by being corrected by the correction unit 161. In doing so, the image quality in the high-quality area can be improved.

  Further, in the image processing unit 151, the correction unit 161 is arranged between the image division unit 51 and the image storage unit 52, so that the correction unit 161 has the low image quality image data supplied from the image division unit 51. And the corrected low image quality image data can be stored in the image storage unit 52. That is, in this case, the low-quality image data read by the image composition unit 53 is already corrected image data.

  Next, image compression processing by the videophone device 1 will be described with reference to the flowchart of FIG.

  Each of the processes of steps S31 to S34 is the same as each of the processes of steps S11 to S14 in FIG.

  If it is determined in step S34 that the frame is a predetermined frame, the process proceeds to step S35, and the image dividing unit 51 supplies the low-quality image data among the divided image data to the image storage unit 52, and the high-quality image data. Is supplied to the correction unit 161. The image storage unit 52 stores the low image quality image data supplied from the image dividing unit 51.

  For example, in step S <b> 35, the image dividing unit 51 supplies the image data corresponding to the A region 111 to the image storage unit 52 among the divided image data, and supplies the image data corresponding to the B region 112 to the correcting unit 161. To do. The image storage unit 52 stores image data corresponding to the A area 111 supplied from the image dividing unit 51. That is, the image data corresponding to the A area 111 stored in the image storage unit 52 corresponds to the A area 111 of the first frame (for example, the frame 1-1 in FIG. 5) among the frames displayed for one second. Only the image data to be stored is stored.

  On the other hand, if it is determined in step S34 that the frame is not a predetermined frame, the process of step S35 is skipped and the process proceeds to step S36. The correction unit 161 reads the low-quality image data from the image storage unit 52, and Each image data is corrected based on the image quality image data and the high quality image data supplied from the image dividing unit 51. The correction unit 161 supplies the corrected high-quality image data and low-quality image data to the image composition unit 53.

  For example, in step S <b> 36, the correction unit 161 reads the image data corresponding to the A region 111 from the image storage unit 52, and stores the image data corresponding to the read A region 111 and the B region 112 supplied from the image dividing unit 51. Based on the corresponding image data, correction is performed so as to reduce the data amount of the image data corresponding to the A area 111, and the image data corresponding to the corrected A area 111 and the image data corresponding to the B area 112 are converted into the image composition unit 53. To supply.

  For example, in step S <b> 36, the correction unit 161 reads the image data corresponding to the A region 111 from the image storage unit 52, the image data corresponding to the read A region 111, and the B region supplied from the image dividing unit 51. Based on the image data corresponding to 112, correction is performed so as to reduce the data amount of the image data corresponding to the A region 111, and the B region 112 according to the reduced data amount of the image data corresponding to the A region 111. May be corrected so as to increase the amount of image data corresponding to A, and the corrected image data corresponding to the A area 111 and image data corresponding to the B area 112 may be supplied to the image composition unit 53.

  Here, as a correction method, for example, correction is performed so that a predetermined bit has a fixed value out of the bit length of each pixel of the image displayed on the screen. Specifically, in the example shown in FIG. 6, assuming that the bit length of each pixel displayed on the screen 101 is, for example, 8 bits, the A region 111 is a background or the like, and looks like a still image at first glance. In reality, the lower 1 or 2 bits of the 8 bits are constantly changing due to errors or noise due to quantization, so the lower 1 or 2 bits are fixed values. By doing so, it is possible to improve the image quality of the B region 112 by suppressing the generation of difference data due to errors or noise due to quantization, reducing the data amount of the A region 111, and so on.

  Further, since the data amount of the A area 111 is reduced in order to improve the image quality of the B area 112, the image quality of the A area 111 may deteriorate in resolution such as color and brightness. Since this is an area that does not have high image quality, there is no practical problem.

  Note that the correction method is not limited to the above-described method of correcting the predetermined bits so as to be a fixed value, and may be any method that can reduce the data amount of the low-quality image data (A area 111).

  Each of the processing of step S38 and step S39 is the same as each of the processing of step S17 and step S18 of FIG.

  In this way, one frame is divided into a high image quality area and a low image quality area, and among the frames displayed in one second, in the low image quality data corresponding to the low image quality area, the same image data is stored. In addition, by correcting so that the data amount of the low-quality image data is reduced, the image quality of the high-quality area can be improved when the image data is encoded. That is, according to the present invention, the image quality can be maintained even when the moving image compression rate is lowered, and the image quality can be improved without increasing the moving image compression rate.

  By the way, although it is area | region information mentioned above, area | region information can also be changed dynamically besides the method of specifying based on the data set previously mentioned above. Hereinafter, processing for dynamically changing the region information will be described with reference to FIGS. 9 to 15.

  First, referring to FIG. 9 to FIG. 12, by designating a moving part of an image as a high-quality area and a stationary part as a low-quality area, a high-quality area and a low-quality area (area) are designated. A process for dynamically changing (information) will be described.

  FIG. 9 is a block diagram illustrating a detailed configuration of the image processing unit 171. The same parts as those shown in FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image processing unit 171 is provided instead of the image processing unit 14 of FIG. The image processing unit 171 sequentially acquires the image data supplied from the A / D conversion unit 13 under the control of the CPU 16 and captures one frame of the acquired image data. The image processing unit 171 executes predetermined processing such as processing for dividing the captured image data into image data in an area where high image quality is to be achieved and image data in an area where high image quality is not high, for example. The processed image data is supplied to the encoder 15.

  The image processing unit 171 is configured to include an image dividing unit 51, an image storage unit 52, an image composition unit 53, a motion amount detection unit 181, and a region determination unit 182.

  The motion amount detection unit 181 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. The motion amount detection unit 181 detects the movement amount (motion amount) of the subject (object) from the captured image data for one frame, and supplies the detected motion amount to the region determination unit 182. For example, the motion amount detection unit 181 detects a motion vector (motion amount) from the image data supplied from the A / D conversion unit 13 by block matching (algorithm), and the detected motion vector (motion amount) is a region. It supplies to the determination part 182.

  The region determination unit 182 sets (determines) a high image quality region and a low image quality region based on the motion amount supplied from the motion amount detection unit 181. The region determination unit 182 supplies region information, which is information indicating the determined high-quality region and low-quality region, to the image dividing unit 51. For example, based on the motion amount supplied from the motion amount detection unit 181, the region determination unit 182 sets the image of the region where the motion amount is detected as a high-quality region, and reduces the image of the region where the motion amount is not detected. The image quality area is determined, and the determined area information is supplied to the image dividing unit 51.

  The image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. The image dividing unit 51 divides the captured image data for one frame into high-quality data and low-quality data in accordance with the region information supplied from the region determining unit 182. Of the divided image data, the image dividing unit 51 supplies high-quality image data to the image composition unit 53 and supplies low-quality image data to the image storage unit 52.

  Next, image compression processing by the videophone device 1 will be described with reference to the flowchart of FIG.

  Each of the processing of step S51 and step S52 is the same as each of the processing of step S11 and step S12 of FIG. 4, and description thereof will be omitted.

  In step S <b> 53, the motion amount detection unit 181 detects the motion amount from the image data supplied from the A / D conversion unit 13 and supplies the detected motion amount to the region determination unit 182.

  For example, in step S53, the motion amount detection unit 181 sequentially acquires the image data supplied from the A / D conversion unit 13, captures one frame of the acquired image data, and captures one frame. The movement amount (motion amount) of the subject (object) is detected from the image data of the minute, and the detected movement amount is supplied to the region determination unit 182.

  For example, in step S53, the motion amount detection unit 181 detects a motion vector (motion amount) from the image data supplied from the A / D conversion unit 13 by block matching (algorithm), and the detected motion vector (motion Amount) is supplied to the region determination unit 182. Specifically, the motion amount detection unit 181 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. Set a macroblock of a predetermined size (for example, vertical 16 (dots) x horizontal 16 (dots)), and by matching the block with the previous frame, the upper left macroblock on the frame starts at the right The absolute difference sum (evaluation value) is calculated sequentially in order up to the lower macroblock, and the motion of the place where the evaluation value is minimum is detected as a motion vector.

  The motion amount detection unit 181 detects the motion vector (motion amount) from the image data supplied from the A / D conversion unit 13, but the captured frame is divided into a plurality of regions and the divided regions are used. You may make it detect a motion vector (motion amount) for every. Further, the motion vector (motion amount) is not limited to block matching, and may be detected by a method such as a gradient method, for example.

  In step S54, the region determination unit 182 sets (determines) a high image quality region and a low image quality region based on the motion amount supplied from the motion amount detection unit 181. The region determination unit 182 supplies region information, which is information indicating the determined high-quality region and low-quality region, to the image dividing unit 51.

  For example, in step S54, the region determination unit 182 uses the motion amount supplied from the motion amount detection unit 181 as an image of the region in which the motion amount is detected as a high-quality region, and the region in which the motion amount is not detected. Are determined as low-quality areas, and the determined area information is supplied to the image dividing unit 51. Specifically, in step S54, the region determination unit 182 determines, based on the motion amount supplied from the motion amount detection unit 181, 5% around the image in which no motion amount is detected as a low image quality region, The other area in which the amount of motion is detected is determined as the high image quality area, and the determined “5% around the image is set as the low image quality area (A area 111) and the other images are defined as the high image quality area (B area 112). Is supplied to the image dividing unit 51.

  Here, with reference to FIG. 11 and FIG. 12, details of the image quality region determined by the region determination unit 182 will be described.

  FIG. 11 is a diagram illustrating an example of a screen of a display unit (not shown) that displays an image captured by the camera 11. The same parts as those shown in FIG. 6 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  In the example illustrated in FIG. 11, the region determination unit 182 determines that the 5% of the periphery of the image displayed on the screen 101 as the A region 111 as an image (for example, a background) of a low-quality region in which the amount of motion is not detected. The image of the high-quality area where the amount of motion other than that is detected (for example, the user's face or body) is determined as the B area 112. That is, an area having a resolution of vertical 432 (dots) × horizontal 576 (dots) excluding 5% around the image displayed on the screen 101 is the B area 112.

  That is, the proportion of the B area 112 in the image displayed on the screen 101, that is, the resolution of the image displayed on the screen 101: the resolution of the B area 112 is 480 vertical (dots) × 640 horizontal (dots): 432 vertical. Since it is (dots) x horizontal 576 (dots), it is 81%. Similarly, the proportion of the area A 111 in the image displayed on the screen 101 is 19%.

  In this case, for example, when there are many moving image areas such as when the user is operating, the moving image area of the screen 101 is 81%, and the stationary image area is 19%. %. Therefore, for example, compared to the case where 40% of the periphery of the image displayed on the screen 101 is set as the A area 111 and the other area is set as the B area 112 (for example, FIG. 12 described later), Although the amount of allocated data is reduced, the image quality of a wider area can be improved.

  In addition, the image quality region determined by the region determination unit 182 changes in conjunction with the amount of motion detected by the motion amount detection unit 181, and thus changes continuously in the case of a moving image. Next, with reference to FIG. 12, the image quality area when the number of user operations is small compared to FIG. 11 will be described.

  FIG. 12 is a diagram illustrating an example of a screen of a display unit (not shown) that displays an image captured by the camera 11. The same parts as those shown in FIG. 6 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  In the example illustrated in FIG. 12, the area determination unit 182 uses the A area 111 to obtain 40% of the periphery of the image displayed on the screen 101 as an image (for example, a background) of a low image quality area in which the amount of motion is not detected. In other words, an image in a high-quality area (for example, a user's face or body) from which the amount of motion other than that is detected is determined as the B area 112. That is, an area having a resolution of 288 vertical (dots) × 384 horizontal (dots) excluding 40% of the periphery of the image displayed on the screen 101 is the B area 112.

  That is, the ratio of the B area 112 in the image displayed on the screen 101, that is, the resolution of the image displayed on the screen 101: the resolution of the B area 112 is 480 vertical (dots) × 640 horizontal (dots): 288 vertical (Dots) x horizontal 384 (dots), so 36%. Similarly, the ratio of the A area 111 in the image displayed on the screen 101 is 64%.

  In this case, for example, when there is no movement of the user, there are few moving image areas, so the moving image area of the screen 101 is 36%, and the stationary image area is 64%. It becomes. Therefore, the amount of data allocated to the B area 112 as compared to the case where 5% around the image displayed on the screen 101 is set as the A area 111 and the other area is set as the B area 112 (for example, FIG. 11). Increases, and the image quality can be improved.

  As described above, when the amount of motion is large, the ratio of the B region 112 to the whole increases. When the amount of motion is small, the ratio of the B region 112 to the whole decreases. ) Can be further efficiently compressed while maintaining the image quality.

  Note that the method of specifying the A region 111 and the B region 112 is not limited to the method of specifying the rectangular (square) region shown in FIGS. 11 and 12, for example, the user's contour displayed on the screen 101, etc. Any method that can specify an area on the screen 101 may be used.

  Returning to the flowchart of FIG. 10, in step S55, the image division unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13, and captures one frame of the acquired image data. The image dividing unit 51 divides the captured image data for one frame into high-quality data and low-quality data in accordance with the region information supplied from the region determining unit 182.

  For example, in step S55, the image dividing unit 51 uses the image data supplied from the A / D conversion unit 13 according to the region information in which 5% around the image supplied from the region determining unit 182 is a low image quality region. The captured image data for one frame is divided into image data corresponding to area A 111 in FIG. 11 and image data corresponding to area B 112 in FIG.

  Further, for example, in step S55, the image dividing unit 51 determines the image supplied from the A / D conversion unit 13 according to the region information in which 40% of the periphery of the image supplied from the region determining unit 182 is a low image quality region. The image data for one frame captured from the data is divided into image data corresponding to area A 111 in FIG. 12 and image data corresponding to area B 112 in FIG.

  Each of the processes in steps S56 to S60 is the same as each of the processes in steps S14 to S18 in FIG.

  In this way, the high-quality area that is the part with movement (the part that is attracting attention) and the low-quality area that is the other part dynamically change according to the movement of the image. The (highlighted part) always has high image quality, and a clearer image can be recognized by the other party watching the video. For example, in the videophone device 1-1, since the image of the user who is making a videophone image picked up by the camera 11 has a high image quality, a clearer user's image is displayed to the other party who uses the videophone device 1-2. Images can be recognized.

  Although it is the timing when the area determining unit 182 determines the image quality area, it is desirable to determine the first frame (for example, the frame 1-1 in FIG. 5) among the frames displayed for one second. That is, low-quality image data read from the image storage unit 52 synthesized by the image synthesis unit 53 and supplied from the image division unit 51 by making the image quality regions the same in the frames displayed per second. It becomes possible to synchronize with the high-quality image data.

  In the above-described example, the region determination unit 182 determines the region in which the amount of motion is detected as the high-quality region, and determines the region in which the amount of motion is not detected as the low-quality region. When the amount of motion exceeds a predetermined threshold, the region that exceeds the threshold is determined as a high-quality region. On the other hand, when the amount of motion is less than the predetermined threshold, You may make it decide to.

  By the way, since the Internet which is an example of the network 2 is a best effort type network, there is a possibility that the network 2 is congested due to a communication amount exceeding the processing capability of the network 2 and congestion occurs. Further, since the state of the network 2 is constantly changing, the communication status between the videophone device 1-1 and the videophone device 1-2 in FIG. 1 is also constantly changing. Therefore, hereinafter, a process of detecting the delay time of the network 2 and dynamically changing the area information according to the detected delay time will be described with reference to FIGS. 13 to 15.

  FIG. 13 is a block diagram illustrating a detailed configuration of the image processing unit 201. The same parts as those shown in FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image processing unit 201 is provided instead of the image processing unit 14 in FIG. The image processing unit 201 sequentially acquires the image data supplied from the A / D conversion unit 13 under the control of the CPU 16 and captures one frame of the acquired image data. For example, the image processing unit 201 executes predetermined processing such as processing to divide the captured image data into image data in an area where high image quality is desired and image data in an area where high image quality is not high. The processed image data is supplied to the encoder 15.

  The image processing unit 201 is configured to include an image dividing unit 51, an image storage unit 52, an image composition unit 53, an RTT (Round-Trip Time) measurement unit 211, and an area determination unit 212.

  The RTT measurement unit 211 generates an RTT measurement packet for detecting a delay time (RTT) of a packet to be transmitted, and supplies the generated RTT measurement packet to the communication unit 19. For example, the RTT measurement unit 211 generates an RTT measurement packet that stores data indicating the transmission time, and supplies the generated RTT measurement packet to the communication unit 19. The RTT measurement unit 211 calculates a delay time based on the RTT measurement packet received by the communication unit 19 and supplied from the communication unit 19. The RTT measurement unit 211 supplies the calculated delay time to the region determination unit 212.

  Further, the RTT measurement unit 211 generates an RTT measurement packet having the same data part as the data part of the RTT measurement packet supplied from the communication unit 19, and supplies the generated RTT measurement packet to the communication unit 19. For example, the RTT measurement unit 211 generates an RTT measurement packet that stores the same time as the transmission time stored in the data part of the RTT measurement packet supplied from the communication unit 19, and transmits the generated RTT measurement packet to the communication unit 19. To supply.

  The area determination unit 212 stores the delay time supplied from the RTT measurement unit 211 in, for example, a memory provided in the area determination unit 212. The region determination unit 212 acquires the delay time supplied from the RTT measurement unit 211 and stored in a memory provided in the region determination unit 212.

  In addition, the area determination unit 212 sets (determines) a high image quality area and a low image quality area based on the acquired delay time. The region determining unit 212 supplies region information indicating the determined high image quality region and low image quality region to the image dividing unit 51. For example, the region determination unit 212 determines region information based on the acquired delay time so that the high-quality region is widened and the low-quality region is narrowed, and the determined region information is supplied to the image dividing unit 51. To do.

  The image dividing unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13 and captures one frame of the acquired image data. The image division unit 51 divides the captured image data for one frame into high-quality data and low-quality data in accordance with the region information supplied from the region determination unit 212. Of the divided image data, the image dividing unit 51 supplies high-quality image data to the image composition unit 53 and supplies low-quality image data to the image storage unit 52.

  Next, image compression processing by the videophone device 1 will be described with reference to the flowchart of FIG.

  Each of the processing of step S71 and step S72 is the same as each of the processing of step S11 and step S12 of FIG. 4, and description thereof will be omitted.

  In step S73, the area determination unit 212 acquires the delay time supplied from the RTT measurement unit 211 and stored by itself. For example, the region determination unit 212 acquires the delay time supplied from the RTT measurement unit 211 and stored in a memory provided in the region determination unit 212.

  Although details will be described later, the delay time supplied from the RTT measurement unit 211 is obtained by repeating transmission and reception of the RTT measurement packet between the videophone device 1-1 and the videophone device 1-2. Calculated by

  In step S74, the region determination unit 212 sets (determines) a high image quality region and a low image quality region based on the acquired delay time. The region determining unit 212 supplies region information indicating the determined high image quality region and low image quality region to the image dividing unit 51.

  For example, when the acquired delay time exceeds a predetermined time in step S74 (for example, when congestion occurs in the network 2), the area determination unit 212 has a communication amount that exceeds the processing capability of the network 2. Since it occurs and the network 2 is congested, it may take a long time to transmit the image data, or a part of the image data may be discarded and not reach the other party. The area information is determined so that the image quality can be surely reached to the other party and the low image quality area that does not need to improve the image quality is widened, and the determined area information is supplied to the image dividing unit 51.

  More specifically, in step S74, the region determination unit 212 determines 40% of the periphery of the image as a low image quality region based on the acquired delay time, and determines other regions as high image quality regions (for example, , The determined area information is supplied to the image dividing unit 51.

  That is, by reducing the data amount (traffic) of the image data, for example, even when congestion occurs in the network 2, the image data in the high-quality area can be transmitted to the other party more reliably.

  On the other hand, for example, if the acquired delay time is equal to or shorter than a predetermined time in step S74 (for example, when congestion does not occur in the network 2), the area determination unit 212 has a margin for traffic flowing through the network 2. Since the image data can be transmitted to the other party without fail, it is possible to widen the high-quality area, which is a more important area, so that a clear image can reach the other party. The region information is determined so as to narrow the good low image quality region, and the determined region information is supplied to the image dividing unit 51.

  More specifically, in step S74, the region determination unit 212 determines, based on the acquired delay time, 5% around the image as a low image quality region, and determines other regions as high image quality regions (for example, , And the determined area information is supplied to the image dividing unit 51.

  That is, for example, when congestion does not occur in the network 2, a clearer image can be delivered to the other party by increasing the amount of data in the high-quality area.

  In step S75, the image division unit 51 sequentially acquires the image data supplied from the A / D conversion unit 13, and captures one frame of the acquired image data. The image division unit 51 divides the captured image data for one frame into high-quality data and low-quality data in accordance with the region information supplied from the region determination unit 212.

  For example, in step S <b> 75, the image dividing unit 51 uses the image data supplied from the A / D conversion unit 13 according to the region information in which 5% around the image supplied from the region determining unit 212 is a low image quality region. The captured image data for one frame is divided into image data corresponding to area A 111 in FIG. 11 and image data corresponding to area B 112 in FIG.

  Further, for example, in step S75, the image dividing unit 51 determines the image supplied from the A / D conversion unit 13 according to the region information in which 40% of the periphery of the image supplied from the region determining unit 212 is a low image quality region. The image data for one frame captured from the data is divided into image data corresponding to area A 111 in FIG. 12 and image data corresponding to area B 112 in FIG.

  Each of the processes in steps S76 to S80 is the same as each of the processes in steps S14 to S18 in FIG. 4, and the description thereof is omitted.

  Thus, for example, the Internet, which is an example of the network 2, is a best-effort network, and therefore, a communication amount exceeding the processing capability of the network may occur, the network may be congested, and congestion may occur. According to the present invention, the delay time of the network is detected, and the high image quality region and the low image quality region are determined according to the delay time. Therefore, even if congestion occurs in the network 2, an image of an important region can be obtained. Can be displayed more reliably.

  Although it is the timing when the area determining unit 212 determines the image quality area, it is desirable to determine the first frame (for example, the frame 1-1 in FIG. 5) among the frames displayed for one second. That is, low-quality image data read from the image storage unit 52 synthesized by the image synthesis unit 53 and supplied from the image division unit 51 by making the image quality regions the same in the frames displayed per second. It becomes possible to synchronize with the high-quality image data.

  Next, with reference to the flowchart of FIG. 15, the process of transmitting and receiving the RTT measurement packet by the videophone device 1-1 and the videophone device 1-2 will be described.

  In step S101, the RTT measurement unit 211 of the videophone device 1-1 determines whether or not a predetermined time has elapsed by determining whether or not the built-in timer has expired. For example, in step S101, the RTT measurement unit 211 determines whether or not a predetermined time has elapsed by comparing a timer value with a predetermined value such as 10 (s).

  If it is determined in step S101 that the predetermined time has not elapsed, the process returns to step S101 and the above-described processing is repeated. That is, although details will be described later, the RTT measurement unit 211 of the videophone device 1-1 generates and transmits an RTT measurement packet at a certain interval (for example, 10 (s)).

  If it is determined in step S101 that the predetermined time has passed, the process proceeds to step S102, where the RTT measurement unit 211 of the videophone device 1-1 generates an RTT measurement packet, and the generated RTT measurement packet is transmitted to the videophone. It transmits to the communication part 19 of the apparatus 1-1. For example, the RTT measurement unit 211 obtains the current time (the transmission time of the RTT measurement packet) from the built-in RTC, generates an RTT measurement packet that stores the acquired transmission time, and transmits the generated RTT measurement packet to the communication unit 19 is supplied.

  In step S <b> 103, the communication unit 19 of the videophone device 1-1 transmits the RTT measurement packet supplied from the RTT measurement unit 211 to the videophone device 1-2 via the network 2.

  In step S111, the communication unit 19 of the videophone device 1-2 receives the RTT measurement packet transmitted from the videophone device 1-1 via the network 2, and uses the received RTT measurement packet as the videophone device 1. -2 to the RTT measuring unit 211.

  In step S112, the RTT measurement unit 211 of the videophone device 1-2 generates an RTT measurement packet having the same data portion as the data portion of the RTT measurement packet supplied from the communication unit 19 of the videophone device 1-2. The generated RTT measurement packet is supplied to the communication unit 19 of the videophone device 1-2. For example, in step S112, the RTT measurement unit 211 generates an RTT measurement packet that stores the same time as the transmission time stored in the data part of the RTT measurement packet supplied from the communication unit 19, and generates the generated RTT measurement packet. Is supplied to the communication unit 19.

  In step S113, the communication unit 19 of the videophone device 1-2 sends the RTT measurement packet supplied from the RTT measurement unit 211 of the videophone device 1-2 to the videophone device 1-2 via the network 2. Send.

  As described above, when receiving the RTT measurement packet, the videophone device 1-2 returns the RTT measurement packet to the videophone device 1-1 immediately.

  In step S104, the communication unit 19 of the videophone device 1-1 receives the RTT measurement packet transmitted from the videophone device 1-2 via the network 2, and uses the received RTT measurement packet as the videophone device 1. −1 to the RTT measuring unit 211.

  In step S105, the RTT measurement unit 211 of the videophone device 1-1 is received based on the RTT measurement packet received by the communication unit 19 of the videophone device 1-1 and supplied from the communication unit 19 of the videophone device 1-1. Then, the delay time is calculated. The RTT measuring unit 211 of the videophone device 1-1 supplies the calculated delay time to the area determining unit 212 of the videophone device 1-1.

  For example, in step S <b> 105, the RTT measurement unit 211 transmits from the videophone device 1-2 according to the RTT measurement packet received and received by the communication unit 19 and supplied from the communication unit 19. Based on the received RTT measurement packet, the delay time is calculated, and the calculated delay time is supplied to the region determination unit 212.

  Specifically, the RTT measurement packet stores a transmission time indicating the time when the user himself / herself was transmitted from the videophone device 1-1, and the delay time is calculated based on this transmission time. That is, the RTT measurement unit 211 of the video phone device 1-1 is based on the time when the RTT measurement packet is transmitted from the video phone device 1-1 and the time when the RTT measurement packet is received from the video phone device 1-2. Calculate the delay time. For example, in step S <b> 105, the RTT measurement unit 211 of the videophone device 1-1 calculates the delay time using Expression (1).

  (Reception time of RTT measurement packet) − (Transmission time of RTT measurement packet) (1)

  Here, the reception time of the RTT measurement packet is the time acquired from the built-in RTC (Real Time Clock) when the RTT measurement unit 211 of the videophone device 1-1 receives the RTT measurement packet. The transmission time of the RTT measurement packet is the transmission time of the RTT measurement packet stored in the RTT measurement packet received from the videophone device 1-2.

  In step S106, the area determination unit 212 of the videophone device 1-1 uses the delay time supplied from the RTT measurement unit 211 of the videophone device 1-1, for example, the area determination unit 212 of the videophone device 1-1. The process ends after storing the data in a memory provided inside.

  As described above, in addition to the image compression processing described with reference to the flowchart of FIG. 14, the video phone apparatus 1-1 and the video phone apparatus 1-2 are described with reference to the flowchart of FIG. 15. By repeatedly executing the process of transmitting and receiving the RTT measurement packet, it is possible to monitor the delay time of the network 2 and determine in real time whether or not congestion occurs in the network 2. Note that a packet loss may be detected, and the area information may be changed accordingly.

  In the above-described example, the videophone device 1 has been described. However, the present invention is not limited thereto, and for example, an antenna and a tuner may be provided instead of the camera 11 and the input terminal 12 in FIG. It is. In this case, the tuner is a broadcast signal corresponding to a broadcast wave of a television broadcast such as a terrestrial wave or a radio wave from a broadcast satellite received from an antenna according to a designation of a broadcast station to be received according to a user operation, The broadcast signal supplied from the antenna is demodulated, and the image data obtained by the demodulation is supplied to the A / D converter 13.

  Further, for example, a configuration in which a recording medium and a drive are provided instead of the camera 11 and the input terminal 12 of FIG. 2 may be employed. In this case, for example, when a recording medium such as a CD (Compact Disc), a DVD, or a Blu-ray Disc is loaded, the drive drives them and acquires programs and data recorded there. The drive supplies the acquired program and data to the A / D converter 13. Furthermore, for example, a VTR (Videotape Recorder) and an input terminal may be provided instead of the camera 11 and the input terminal 12 of FIG. In this case, the input terminal supplies the image data supplied from the VTR to the A / D converter 13.

  In other words, the present invention is not limited to the camera 11 and the input terminal 12 described above, and any configuration that can supply image data to the A / D converter 13 may be used. Further, when the image data supplied from the outside is digital data, the image data may be supplied to the image processing unit 14 without using the A / D conversion unit 13.

  Furthermore, for example, a configuration in which a drive and a recording medium are provided instead of the communication unit 19 in FIG. In that case, for example, when a recording medium such as a CD, a DVD, or a Blu-ray Disc is loaded, the drive drives them to record the image data supplied from the CPU 16 on the recording medium. That is, the present invention is not limited to the communication unit 19 described above, and any configuration that can output image data from the CPU 16 may be used.

  In the above-described example, a plurality of embodiments are described. However, each embodiment may be implemented alone, and a plurality of embodiments may be combined. Furthermore, the above-described frame rate or bit rate can be changed to an arbitrary value.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 16 is a block diagram showing an example of the configuration of the videophone device 1 that executes the above-described series of processing by a program. The CPU 301 executes various processes according to programs recorded in the ROM 302 or the recording unit 308. The RAM 303 appropriately stores programs executed by the CPU 301 and data. The CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.

  An input / output interface 305 is also connected to the CPU 301 via the bus 304. Connected to the input / output interface 305 are an input unit 306 including a keyboard, a mouse, and a microphone, and an output unit 307 including a display and a speaker. The CPU 301 executes various processes in response to commands input from the input unit 306. Then, the CPU 301 outputs an image or the like obtained as a result of the processing to the output unit 307.

  The recording unit 308 connected to the input / output interface 305 includes, for example, a hard disk, and records programs executed by the CPU 301 and various data. A communication unit 309 communicates with an external device via the Internet or other networks. In this example, the communication unit 309 operates as an interface with the outside that acquires an input image or outputs an output image.

  Alternatively, the program may be acquired via the communication unit 309 and recorded in the recording unit 308.

  The drive 310 connected to the input / output interface 305 drives the magnetic disk 321, the optical disk 322, the magneto-optical disk 323, the semiconductor memory 324, or the like when they are loaded, and programs and data recorded there. Get etc. The acquired program and data are transferred to the recording unit 308 and recorded as necessary.

  As shown in FIG. 16, a recording medium storing a program for performing a series of processing is distributed to provide a program to a user separately from a computer. Disk), optical disk 322 (including CD-ROM (Compact Disc-Read Only Memory), DVD), magneto-optical disk 323 (including MD (Mini-Disc) (trademark)), or semiconductor memory 324. In addition to a package medium, the program is configured by a ROM 302 in which a program is recorded and a hard disk included in the recording unit 308, which is provided to a user in a state of being preinstalled in a computer.

  The program for executing the series of processes described above is installed in a computer via a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary. You may be made to do.

  Further, in the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in time series according to the described order, but is not necessarily performed in time series. It also includes processes that are executed individually.

It is a block diagram which shows the structure of one embodiment of a videophone system. It is a block diagram which shows the structure of one Embodiment of the video telephone apparatus to which this invention is applied. It is a block diagram explaining the detailed structure of an image process part. It is a flowchart explaining the process of image compression. It is a figure explaining the detail of a frame. It is a figure which shows the example of the screen which displays the image imaged with a camera. It is a block diagram explaining the other detailed structure of an image process part. It is a flowchart explaining the process of image compression. It is a block diagram explaining the other detailed structure of an image process part. It is a flowchart explaining the process of image compression. It is a figure which shows the example of the screen of the display part which displays the image imaged with a camera. It is a figure which shows the example of the screen of the display part which displays the image imaged with a camera. It is a block diagram explaining the other detailed structure of an image process part. It is a flowchart explaining the process of image compression. It is a flowchart explaining the transmission / reception process of the RTT measurement packet by a video telephone apparatus. It is a block diagram explaining the structure of a video telephone apparatus.

Explanation of symbols

  1, 1-1, 1-2 videophone device, 2 network, 11 camera, 12 input terminal, 13 A / D conversion unit, 14 image processing unit, 15 encoder, 16 CPU, 17 ROM, 18 RAM, 19 communication unit , 51 Image segmentation unit, 52 Image storage unit, 53 Image composition unit, 151 Image processing unit, 161 Correction unit, 171 Image processing unit, 181 Motion amount detection unit, 182 Region determination unit, 201 Image processing unit, 211 RTT measurement unit , 212 area determination unit, 301 CPU, 306 input unit, 307 output unit, 308 recording unit, 309 communication unit, 321 magnetic disk 321, 322 optical disk, 323 magneto-optical disk, 324 semiconductor memory

Claims (6)

A division step of dividing each of the first image data for displaying each of the pictures for displaying the moving image into the first data and the second data according to a predetermined area on the picture;
Storing the second data divided from the first image data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period;
The predetermined period of time is determined from each of the first data divided from each of the first image data of the predetermined number of the pictures of the period and the recorded second data. Generating each of the second image data for displaying each of the given number of pictures;
An image processing method comprising: an encoding step of encoding the second image data by an encoding method for compression encoding based on a difference between pictures. The correction step of correcting the second data so as to replace a lower-order bit value of a predetermined number of digits of the pixel value of the second data with a predetermined value. 2. The image processing method according to 1. A detection step of detecting a movement amount that is a movement amount of an image of a subject in the picture;
2. The method according to claim 1, further comprising: a determination step for determining the region for dividing the first image data into the first data and the second data based on the amount of movement. An image processing method described in 1. A measuring step for measuring a delay time in a network for transmitting the image data to the other party;
2. The method according to claim 1, further comprising: a determination step for determining the region for dividing the first image data into the first data and the second data based on the delay time. An image processing method described in 1. Dividing means for dividing each of the first image data for displaying each of the pictures for displaying the moving image into first data and second data according to a predetermined area on the picture;
Storage means for storing the second data divided from the first image data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period;
The predetermined period of time is determined from each of the first data divided from each of the first image data of the predetermined number of the pictures of the period and the recorded second data. Generating means for generating each of the second image data for displaying each of the number of said pictures;
An image processing apparatus comprising: an encoding unit that encodes the second image data by an encoding method that performs compression encoding based on a difference between pictures. A division step of dividing each of the first image data for displaying each of the pictures for displaying the moving image into the first data and the second data according to a predetermined area on the picture;
Storing the second data divided from the first image data of a predetermined picture out of a predetermined number of pictures displayed in a predetermined period;
The predetermined period of time is determined from each of the first data divided from each of the first image data of the predetermined number of the pictures of the period and the recorded second data. Generating each of the second image data for displaying each of the given number of pictures;
A program for causing a computer to execute an encoding step of encoding the second image data by an encoding method for compression encoding based on a difference between pictures.

Images