JP3927606B2 - Image communication apparatus and system, image receiving apparatus and received image data processing method - Google Patents

Description

[0001]
[Industrial application fields]
The present invention relates to an image communication apparatus and system. To For example, an image communication apparatus and system for transmitting and receiving image data and the like via a communication line. To Related.
[0002]
[Prior art]
In the case of a conventional analog telephone line, for example, a telephone can only transmit voice, and an image communication apparatus such as a facsimile apparatus can transmit image data only at a low speed.
[0003]
However, with recent advances in communication technology, semiconductor technology, and optical technology, digital lines have been established, enabling high-speed and large-capacity data transmission.
[0004]
In particular, the characteristics of digital transmission include the same level of quality without degradation due to transmission, and the integration of media without requiring a transmission path according to the characteristics of the media of the transmission data. Transmission between composite media terminals is now possible. Therefore, in addition to conventional telephones that transmit only voice, telephone terminals that can simultaneously transmit video, so-called TV telephone devices, have appeared.
[0005]
Under these circumstances, international standardization by CCITT (International Telegraph and Telephone Consultative Committee) etc. is being promoted so that mutual communication between different composite terminals is possible, such as videophones and videoconferencing systems using digital lines. The service specifications, protocol specifications, and multimedia multiplexed frame configuration specifications for AV services as AV (Audio Visual) services of CCITT Recommendation (or Draft) 320, H.I. 242, H.M. It is announced as 221 etc.
[0006]
H. 221 exchanges frame configurations and terminal capabilities in AV services from 64 Kbps to 1920 Kbps, and FAS (Fram e Code allocation of Alignment Signal) and BAS (Bit Allocation Signal) is defined. H. Protocols such as capability exchange and communication mode switching between AV terminals using BAS are defined in 242, and a system overview of AV services in general is defined in H320.
[0007]
In the above recommendation (or draft recommendation), after setting up an end-to-end physical connection and establishing synchronization by FAS in the in-channel, a terminal capability exchange sequence using the BAS in the in-channel, and a mode switching sequence by specifying the communication mode A method for performing multimedia communication such as images, sounds, and data between terminals is defined by such procedures.
[0008]
However, it is out of the specified range to change the terminal capability of each terminal according to the situation and which communication mode is used within the range of the converted capability.
[0009]
The information transfer speed of each medium in multimedia communication that simultaneously communicates images, sounds, and data is determined by designating the sound encoding method for sound information, and whether or not the data information is used and transferred when used. The remaining information obtained by subtracting the transfer speed of the audio information and the transfer speed of the data information from the set information transfer speed of the entire communication path is the transfer speed of the image information.
[0010]
[Problems to be solved by the invention]
However, according to the above-described conventional TV telephone apparatus, since the transmission image format for encoding is defined, the image format cannot be changed according to the application, for example.
[0011]
For example, ITU-TS (formerly CCITT) H.264. For the H.261 recommendation, there are only two types of prescribed formats, CIF (Common Intermediate Format) and QCIF (Quarter CIF). High-definition However, there is a problem that only two types of resolutions such as a good image quality can be switched, and the optimum resolution cannot be variably set.
[0012]
Also, when the input image format of the image to be transmitted and the specified format do not match, the input image format is enlarged or reduced to match the size of the specified format, or the received specified format and the output image format are When it does not fit, the specified image format must be enlarged or reduced in order to match the specified format to the output image format. Therefore, there are problems such as requiring useless scaling processing and causing deterioration in image quality due to scaling processing.
[0013]
For example, when transmitting and receiving in a format in which a moving image is inserted into a still image and displaying a screen, the transmitting side first transmits the still image in a specified format, and then transmits the moving image in a specified format. That is, two frames are transmitted, and the reception side generates an image for one frame by performing a process of reducing and scaling the received moving image frame and inserting it into the previously received still image. .
[0014]
Therefore, in order to solve the above-described problems, the present invention provides an image communication apparatus and system that can divide an image flexibly and transmit and receive a moving image at an arbitrary size even if the image format is fixed. Of For the purpose of provision.
[0015]
[Means for Solving the Problems]
In order to achieve the above-described object, the present invention has the following configuration.
[0016]
That is, the image communication apparatus according to the present invention includes one screen. Image Send Be done An area designating unit for designating a first required area and a desired second required area in the first required area as a moving image area; Of the second required area Video and , The first required area excluding the second required area Combination means for combining still images, encoding means for encoding the image data combined by the combination means, and image data encoded by the encoding means When, The first required area specified by the area specifying means The region position information and the position of the region of the moving image within the first required region Area of the second required area position information When Transmission means for transmitting the image to another image communication apparatus.
[0018]
Furthermore, encoded image data When, Said first required area Region position information and Area of the second required area position information When Receiving means for receiving the image data, decoding means for decoding the image data received by the receiving means, and the first required area Region position information and Area of the second required area position Based on the information, the moving image of the second required area is display In order to perform processing, the image processing apparatus includes extraction means for extracting a moving image from the image data decoded by the decoding means.
[0019]
For example, the encoding unit encodes image data in units of blocks, and the region specifying unit specifies regions in units of blocks.
[0020]
For example, the combination means combines different moving images when at least two areas are designated by the area designation means.
[0021]
For example, the still image is a part of the moving image.
[0023]
For example, the encoding means performs encoding using inter-frame differences.
[0024]
A speech processing unit that performs speech processing; a speech encoding unit that encodes speech data processed by the speech processing unit; and a speech transmission unit that transmits speech data encoded by the speech encoding unit; It is characterized by having.
[0025]
Further, in the image communication system of the present invention, the first image communication apparatus has one screen. Image Send Be done An area designating unit for designating a first required area and a desired second required area in the first required area as a moving image area; Of the second required area Video and , The first required area excluding the second required area Combination means for combining still images, encoding means for encoding the image data combined by the combination means, and image data encoded by the encoding means When, The first required area specified by the area specifying means The region position information and the position of the region of the moving image within the first required region Area of the second required area position information When Transmitting means for transmitting the image data to another image communication device, wherein the second image communication device is configured to transmit the encoded image data. When, Said first required area Region position information and Area of the second required area position information When Receiving means for receiving the image data, decoding means for decoding the image data received by the receiving means, and the first required area Region position information and Area of the second required area position Based on the information, the moving image of the second required area is display In order to perform processing, the image processing apparatus includes extraction means for extracting a moving image from the image data decoded by the decoding means.
[0026]
[Action]
In the above configuration, it is possible to encode and transmit more efficiently. Further, it is possible to obtain a specific effect that a plurality of received images can be extracted and selectively displayed or simultaneously displayed.
[0027]
【Example】
Hereinafter, an embodiment according to the present invention will be described in detail with reference to the drawings.
[0028]
FIG. 1 is a block diagram showing the configuration of the TV telephone apparatus of this embodiment. In FIG. 1, 1 is a microphone, 2 is a speaker, 3 is a gain adjustment function for adjusting a volume level, and an echo is erased when the microphone 1 and the speaker 2 are used as audio input / output means, according to an instruction from the system control unit 14. A voice processing unit having an echo canceling function, a dial tone, a ringing tone, a busy tone, a tone generation function such as a ringing tone, and the like, 4 is 64 kbps PCM (A-law), 64 kbps PCM (μ-law) according to an instruction from the system control unit ), 7 kHz audio (SB-ADPCM), 32 kbps ADPCM, 16 kbps (for example, APC-AB) 8 kbps, etc., according to a voice encoding / decoding algorithm, a function of A / D converting and encoding a transmission voice signal, a received voice signal Codec with the function of decoding and D / A conversion , 5 is a standard camera that captures a person, 6 is a monitor that displays an input image from the camera 5, a received image from the partner device, and an image from the system control unit 14. An image processing unit for processing, 8 is a transmission image editing unit, and 9 is a reception image editing unit. The detailed configuration of the transmission image editing unit 8 is shown in FIG. 2 described later, and the detailed configuration of the reception image editing unit 9 is shown in FIG. 3 described later. Reference numeral 18 denotes an area discriminating unit, which creates or decodes area information for discriminating various image areas described later in the image processing unit 7.
[0029]
In FIG. 1, reference numeral 10 denotes a specific image generation unit that converts text data or graphic data to be inserted into an image into bitmap data based on a specific image management table in the system control unit 14 and transfers the bitmap data to the image processing unit 7. Reference numeral 11 denotes an image codec unit having a function of encoding a transmission image and a function of decoding a reception image. The image codec unit 11 performs band compression on the raw data of a large-capacity image by various methods such as motion compensation, frame dropping, inter-frame prediction and inter-frame compensation, DCT conversion, vector quantization conversion, and the like. Is reduced in size so that it can be transmitted through a digital line. Currently, the basic interface of the ISDN line is 64 kbps. As an image encoding method that can be transmitted at this transmission speed, H.264 recommended by ITU-TS is used. There are 261.
[0030]
In FIG. 1, 12 multiplexes audio data from the audio codec unit 4, image data from the image codec unit 11, and BAS from the system control unit 14 in units of transmission frames, and receives frames to each medium that is a unit of configuration. This is a demultiplexing unit that separates and notifies each unit. There are 221.
[0031]
13 is a line interface for controlling the line according to the ISDN user network interface, 14 is provided with a CPU, a ROM, a RAM, an auxiliary storage device, a character generator, an image signal generation circuit, etc. A system control unit 15 for performing control of the image, creation of an operation / display screen according to the state, execution of an application program, and the like, 15 is an area designation management table for managing address designation of image memory in the transmission image editing unit 8 and the reception image editing unit 9 , 16 is a specific image management table for managing specific image data, and 17 is an operation unit including a keyboard, a touch panel, and the like used by an operator for inputting area designation information and control information in order to control the apparatus. Each area designation information in this embodiment is input in advance from the operation panel 17 by the operator and stored in the area designation management table 15.
[0032]
Hereinafter, the transmission image editing unit 8 will be described in detail with reference to FIG. FIG. 2 is a block diagram showing a detailed configuration of the transmission image editing unit 8.
[0033]
In FIG. 2, 21 has a function of inputting image data and performing YC separation if it is a composite signal, further demodulating and separating the C signal into Cr and Cb signals, and performing color conversion if RGB signals are necessary. An input processing unit 22 is a scaling processing unit that scales the image format of input image data to a required image format, 23 is a buffer that temporarily stores image data for synchronization, and 24 is a transmission image processing A still image setting unit for setting a still image to be written in the memory 27 according to an instruction from the unit 35, 25 is input image data, still image data from the still image setting unit 24, and specific image data from the image control unit 35 The selection unit selects which data is to be written into the image memory 27. The still image setting unit 24 has a memory for storing various still images in accordance with an instruction from the transmission image control unit 35, and image data can be taken in from the image memory 27 as a still image.
[0034]
Reference numeral 26 denotes a memory write control unit that generates a write address to the image memory 27 and controls the selection unit 25. Reference numeral 27 denotes an image memory. Reference numerals 28, 29, and 30 denote an image memory 27 for the memory write control unit 26. The area designation units a, b, c, 31 for setting the area designation and the data to be selected in the area are memory read control units 32, 33, 34 for generating read address signals from the image memory 27. The area designation units d, e, f, and 35 for setting the area designation to the image memory 27 and the area of the image to be transferred to the image codec unit 11 in the area for the memory read control unit 31 are specific image data and still image The transmission image control for setting the image data and the area designation data to the memory write control unit 26 and the memory read control unit 31 in the area designation units 28 to 30 and 32 to 34. It is a part.
[0035]
Hereinafter, the received image editing unit 9 will be described in detail with reference to FIG. FIG. 3 is a block diagram showing a detailed configuration of the received image editing unit 9.
[0036]
In FIG. 3, reference numeral 41 denotes a memory write control unit that generates a write address signal to the image memory 45, and reference numerals 42, 43, and 44 denote area designation units that set area designation to the image memory 45 for the memory write control unit 41. Reference numerals a, b, c, and 45 denote image memories, 46 denotes a memory read control unit that generates a read address signal from the image memory 45, and 47, 48, and 49 denote areas to the image memory 45 with respect to the memory read control unit 46. The area designating parts d, e, f, 50 for setting the designation are a scaling process part for scaling the image format to a required image format, and 51 is the image data from the scaling process part 50 and the received image control part 54. A combining unit 52 that combines the specific image data with an instruction from the received image control unit 54 controls the combining unit 51 to control the switching timing of the combination. An array controller 53 is a DAC (Digital Analog Converter) that converts the image data output from the combining unit 51 from a digital signal to an analog signal, and 54 is an output of specific image data and the memory write control unit 41 and the memory read control unit 42. This area designation data is set in each of the area designation units 42 to 44, 47 to 49, or over-array switching control to the over-array controller 52 is set.
[0037]
With the configuration described above, the image input to the transmission image area by the transmission image editing unit 8 is inserted into the image before encoding input from the camera 5 shown in FIG. Still image data or specific image data generated by the transmission image control unit 35 can be fitted in part or all.
[0038]
Further, with respect to an image decoded by the image codec unit 11 after reception, the received image editing unit 9 arbitrarily selects a region from the received image region, extracts a part or all of the image, and receives image control The specific image data generated by the unit 54 and the image can be displayed on the monitor 6 in an overlapping manner.
[0039]
2 and the received image control unit 54 shown in FIG. 3 are controlled by the system control unit 14 shown in FIG. 1, and an area designation management table in the system control unit 14 is used. According to the contents of 15, the access to the respective image memories 27 and 45 is controlled.
[0040]
Further, each area information in the transmission image synthesized by the transmission image editing unit 8 and each area information in the reception image received by the reception image editing unit 9 are managed by the area determination unit 18 shown in FIG. The transmission image control unit 35 and the reception image control unit 54 are entrusted with the control. The area information generated in the area discriminating unit 18 is transmitted / received between the TV telephones prior to transmission / reception of the encoded image data.
[0041]
Hereinafter, a specific example of the above-described data fitting process to the transmission image and the received image superimposing process will be described with reference to FIG.
[0042]
In FIG. 4A, the four corners are (x2, y2), (x2, y3), (x3, y2), (x3, y3) from the transmission image control unit 35 to the region designation unit a28, for example. An example is shown in which an area to be displayed is designated and the area is defined as a moving image area. In this case, the memory write control unit 26 switches the selection unit 25 to the input image from the buffer 23 when the access to the image memory 27 reaches the area, and the image memory 27 stores the input image, that is, the moving image. Write the corresponding data. Further, in (a) of FIG. 4, (x0, y0), (x0, y1), (x1, y0), (x1, y1) are transmitted from the transmission image control unit 35 to, for example, another region specifying unit b29. An example in which an area indicated by is designated and the area is defined as a still image area is also shown. When the access to the image memory 27 reaches the area, the memory write control unit 26 switches the selection unit 25 to the still image data from the still image setting unit 24, and the image memory 27 corresponds to the still image data. Write data to be used. In addition, as long as the area | region designation | designated parts 28-30 are not used mutually overlapping, you may use either.
[0043]
FIG. 4B shows an example in which specific image data generated from the transmission image control unit 35 is further inserted into the image shown in FIG. In the area designation of the specific image data, the area designation is first performed for one of the area designation units 28 to 30 as in the above-described example, and when the memory write control unit 26 reaches that area, Data corresponding to the specific image data is written into the image memory 27.
[0044]
The image data generated in the transmission image editing unit 8 as described above is transferred to the image codec unit 11.
[0045]
Next, with reference to FIG. 5, a method for specifying an area in the present embodiment will be described. FIG. 5 is a diagram showing various area designation methods in the present embodiment.
[0046]
FIG. 5A shows that when two people's images are input from the camera 5, only the main image portions of the two people are taken into the image memory 27 as moving images, and the other portions are the same. An example of capturing still image data will be shown. In this case, encoded data is prevented from being generated by the image codec unit 11 in a portion that does not need to be transmitted as a moving image, and the encoded data is allocated only to two important image data whose areas are designated. The transfer efficiency of image data can be improved. Further, by handling these two moving image areas as completely independent images, multiple images can be input from one camera 5.
[0047]
In FIG. 5B, for example, a plurality of cameras 5 a, b, and c are provided, and multi-image input is realized by writing the image input to the image memory 27 while sequentially switching the image input between a, b, and c. An example is shown.
[0048]
As a multi-image input method in this case, a transmission signal control unit 35 generates a synchronization signal and sends it to each camera a to c. Each camera a to c transmits an input image to the transmission image editing unit in accordance with the synchronization. 8 and the transmission image editing unit 8 writes the image data of each camera a to c in the image memory 27 while sequentially switching, and receives all the input images of each camera a to c simultaneously and asynchronously. In this method, the input images of the cameras a to c are sequentially written in the image memory 27 in synchronization with the synchronization generated by the transmission image control unit 35.
[0049]
As in the method described above, the image writing area is designated by setting the area in any of the area designating units 28 to 29 for each camera a to c, and when the processing reaches the set area, The input image from the camera whose area is designated is written into the image memory 27. In this way, it is possible to write input from multiple cameras into one image memory, and write the specific image data or still image data for the remaining unnecessary parts, thereby concentrating the encoding assignment on the necessary parts. Thus, the transfer efficiency of image data can be improved.
[0050]
FIG. 5C shows an example in which a still image is designated as the entire area of the transmission image format, and a moving image input from the camera 5 is designated as a part of the still picture and is inserted.
[0051]
In FIG. 5C, a still image is set by extracting a part of an input image from the camera 5 in the still image setting unit 24. It is of course possible to designate a newly generated still image for the entire area.
[0052]
Also in this case, once the data for the area treated as a still image is transmitted once, difference data as a moving image does not occur after that, so encoding of the transmission image is concentrated only on the moving image portion, and the image data Transfer efficiency is improved.
[0053]
Hereinafter, the encoding process in the image codec unit 11 shown in FIG. 1 will be described with reference to FIG. FIG. 6 is a block diagram showing a detailed configuration for performing the encoding process in the codec unit 11.
[0054]
In FIG. 6, reference numeral 60 denotes an encoding processing unit that performs an encoding process, and reference numeral 61 denotes control of each unit constituting the encoding processing unit 60 or generation of necessary information in accordance with an instruction from the system control unit 14. An encoding control unit 62 is a frame drop processing unit that suppresses the amount of generated code by thinning out image frames to be encoded, and 63 is an inter-frame processing for the same macroblock (hereinafter referred to as MB) of the previous frame and the current frame. INTER / which calculates the mean square error, the variance within the frame, etc., and determines whether the encoding target is the difference value (INTER) between the image of the previous frame and the current frame or the original image value INTRA of the current frame The INTRA discrimination processing unit 64 checks whether the current frame block of interest searches within a certain range of the previous frame and performs pattern matching, that is, motion compensation. (MC) MC discriminating unit for discriminating whether or not to perform MC control by instructing the presence / absence of motion compensation and a motion vector (direction and size) according to the discrimination result of the MC discriminating unit 64 , 66 is an INTER / INTRA processing unit that performs INTER / INTRA processing based on the determination result of the INTER / INTRA determining unit 63, 67 is a filter processing unit that performs filter processing after motion compensation, and 68 is an INTER frame for the same MB. If the error persists, the quantization error accumulates on the receiving side, or a problem such as being unable to recover when a composite image is disturbed due to a transmission error or the like occurs. Therefore, an INTRA frame is generated in a certain period and refreshed. The refresh cycle counter 69 is a threshold for improving the efficiency of data generation when quantized. Quantization threshold value control unit for controlling data such that data below a certain value is set to 0 value so as not to generate data, and 70 is a quantization step size for controlling the quantization step size according to the data accumulation amount of the transmission buffer 81 A control unit 71 is a transmission amount detection unit that detects an accumulation amount of the transmission buffer 81, and 72 is a header information generation unit that generates a frame header, a GOB (Group of Block) header, and an MB header.
[0055]
73 is a frame memory, 74 and 75 are switches for selecting INTER / INTRA processing, 76 is a subtractor that takes the difference between the previous frame and the current frame at the time of INTER processing, and 77 is converted from the spatial domain to the frequency domain by orthogonal transformation. DCT transform (irreversible encoding) unit, 78 is a quantizer, 79 is a VLC (lossless encoding) unit that variable-length encodes the generated data, 80 is a multiplexing unit, 81 is a transmission buffer, and 82 is error correction BCH unit for generating a frame, 83 is an inverse quantizer, 84 is an inverse DCT conversion unit, 85 is an adder for adding the previous frame and the current frame, and 86 and 87 are alternately switched between a read frame and a write frame. Switches, 88 and 89 are frame memories, 90 is an FM controller for controlling the switches 86 and 87 and the frame memories 88 and 89, and 91 is a previous frame. Motion compensation units for detecting pattern matching with the current frame, 92 and 93 are switches for selecting ON / OFF of the filter processing unit 94, 94 is a filter processing unit, and 95 is an input of data to the FIFO memory 96. A switch 96 for turning ON / OFF the FIFO is a FIFO memory.
[0056]
In FIG. 6, the solid line indicates the input / output of the image signal, and the broken line indicates the input / output direction of the control signal.
Next, the configuration of the image data of one screen (one frame) subjected to the encoding process with the configuration shown in FIG. 6 will be described in detail with reference to FIG. FIG. 7A shows an image format example of one screen. Hereinafter, A pixel × B line is represented by A × B.
[0057]
ITU-TS recommendation H. In H.261, since there are a plurality of different standards such as NTSC, PAL, and digital television (HDTV) standards as video signals to be handled, a video signal format common to the world is adopted so that they can communicate with each other. .
[0058]
This is called CIF, and the number of samples is defined as luminance Y of 352 × 288 and color differences Cr and Cb of 176 × 144.
[0059]
As for the sample point (sampling point), as shown in FIG. 7B, the color difference (Cr, Cb) is determined to be an equidistant point of four luminance points (Y1, Y2, Y3, Y4). Yes.
[0060]
Further, a quarter format in which CIF is halved both horizontally and vertically is called QCIF, and the number of samples is defined as 176 × 144 for luminance Y, and 88 × 72 for color differences Cr and Cb.
[0061]
As shown in FIG. 7, each of the CIF and QCIF formats is composed of a plurality of GOB formats, and the GOB format is composed of 33 MB formats. Further, the MB format is composed of four 8 × 8 luminance blocks Y1, Y2, Y3, and Y4 and a total of six blocks of 8 × 8 color difference blocks Cr and Cb, and has a hierarchical structure.
[0062]
The hierarchical structure of the image data shown in FIG. 7 enables encoding to be performed in MB units.
[0063]
In GOB, the number of samples is defined as luminance Y of 176 × 48 and color differences Cr and Cb of 88 × 24, which corresponds to 1/12 of CIF and 1/3 of QCIF. If the CIF is composed of GOB1 to GOB12, the QCIF is composed of GOB1, GOB3, and GOB5.
[0064]
The encoded image data is multiplexed in the multiplexing unit 80 shown in FIG. 6 to have a multiplexed frame configuration. Hereinafter, the frame structure of the encoded image data is shown in FIG. In FIG. 8, for the convenience of description, the description is made with the frame header (FH) added.
[0065]
In FIG. 8, the upper part shows the frame structure by GOB. FH is added to the head of one frame of data, and GOB1 to GOB12 are sequentially transmitted with one block obtained by dividing the screen of one frame into 12 blocks as GOB.
[0066]
Each GOB constituting the frame is divided as shown in the lower part of FIG. The lower part of FIG. 8 shows the detailed configuration of the FH and GOB shown in the upper part.
[0067]
The FH is generated by the header information generation unit 72 shown in FIG. 6 and is composed of PSC, TR, and PTYPE described later. PSC has a frame start code and is 20 bits “0000 0000 0000 0001 0000”. TR is a frame number, and uses a 5-bit value from “1” to “30”. PTYPE is 6-bit type information and includes split screen instruction information, document camera instruction information, screen freeze release, and information source format instruction information (CIF, QCIF).
[0068]
The GOB header is also generated by the header information generation unit 72, and is composed of GBSC, GN, and GQUANT described later. GBSC is a GOB start code and is 16 bits “0000 0000 0000 0001”. GN is a GOB number and uses a 4-bit value from “1” to “12”. When GN is “0”, since it is used as the FH PSC, both the FH PSC and the GOB GBSC + GN can be regarded as continuous values of 20 bits. GQUANT is quantization characteristic information, and includes 5-bit quantization step size information Q determined by the quantization step size control unit 70 shown in FIG.
[0069]
The MB header is also generated by the header information generation unit 72, and is composed of MBA, MTYPE, MQUANT, MVD, and CBP, which will be described later. The MBA is a macroblock address representing the position of the MB, and only the first MB of the 33 MBs constituting one GOB is an absolute value, and the subsequent MBs are variable length codes of the difference. MTYPE is MB type information, and is applied to the MB data such as INTRA (intra-frame coding), INTER (inter-frame differential coding), MC (inter-frame differential coding with motion compensation), FIL (filter), and the like. Indicates the type of processing performed. MQUANT is quantization characteristic information, GQUANT mentioned above With the same information. MVD is motion vector information V. CBP is a significant block pattern having a significant difference value, and includes the number of valid pixel blocks among the four Y blocks and Cr, Cb constituting the MB as information. In the CBP, numbers 1 to 4 are assigned to Y, 5 is assigned to Cb, and 6 is assigned to Cr.
[0070]
The MB header is followed by compression-encoded image data TCOEFF. In the TCOEFF, as described above, the pixel block designated as a significant block in the CBP out of Y4 and Cr, Cb is compressed and entered. An EOB is added to the end of the pixel block.
[0071]
The image data encoded as described above is added with an error correction frame bit in the BCH 82 and is actually transmitted on the line as an error correction frame. The configuration of this error correction frame will be described with reference to FIG.
[0072]
In FIG. 9, in one frame, an error correction frame bit Sn indicating the presence / absence of error correction is 1 bit, a fill identifier Fi is 1 bit indicating whether or not it is moving image data, and image data subjected to BCH encoding is displayed. It consists of 512 bits with 492 bits and error correction parity of 18 bits. Furthermore, one multiframe is composed of 8 frames.
[0073]
The image compression method described above is based on the ITU-TS H.264 standard. If it complies with this recommendation, it is possible to perform mutual communication with other TV phones that comply with this recommendation.
[0074]
Hereinafter, the encoding method in the present embodiment will be briefly described. In the present embodiment, efficient encoding is performed by combining the following methods.
[0075]
(1) Two-dimensional data in a frame is made into an 8 × 8 block by taking advantage of the fact that the correlation between pixels is strong in a natural image, the frequency components are concentrated in a low frequency, and the high frequency is small. Intraframe coding method for DCT conversion.
[0076]
(2) An inter-frame coding method in which a difference between frames is taken in an image block at the same position in the previous frame and the current frame, and an 8 × 8 block is subjected to two-dimensional DCT conversion with respect to the difference value. .
[0077]
(3) When an image block similar to the current frame moves relatively adjacently from the previous frame, this is detected and only the information on the amount and direction of movement of the image block is sent, and the image data itself is not sent. Motion compensation that reduces the amount of generated data.
[0078]
(4) A zero-run length encoding method using the fact that coefficient values for each frequency after DCT conversion are generated in the low frequency region, but the values are less likely to be generated in the high frequency region and the zero value continues.
[0079]
(5) A quantization method for adjusting the data generation amount by changing the data quantization step width according to the data generation amount.
[0080]
(6) A short code value is assigned to a frequently occurring data pattern, and a long code value is assigned to a less frequently occurring data pattern, thereby converting the data amount to a smaller amount than the total generated data amount. A variable length encoding method.
[0081]
(7) A frame dropping method that skips frames and drops image data itself.
[0082]
The plurality of encoding methods described above are effectively used to compress the image data so that moving images can be communicated even in low-rate communication.
[0083]
In this embodiment, the boundaries of the designated areas of the moving image data, the specific image data, and the still image data shown in FIG. 4B are the minimum unit (8 × 8) block boundaries to be encoded. To suit. Therefore, it is possible to specify the above-mentioned various data areas for each block to be encoded. In addition, since the encoding of moving image data and the encoding of still image data including specific image data can be clearly divided into blocks by information such as MTYPE in the MB header as described above, Optimal encoding according to the image becomes possible.
[0084]
In the case of a still image, the first frame is transmitted in the INTRA mode, and the subsequent frames are transmitted in the INTER mode, so that a difference value does not occur in the several frames, and finally the block No longer needs to send data. Therefore, very efficient encoding is possible.
[0085]
Hereinafter, an image communication system using the TV telephone of the present embodiment will be described with reference to FIG. FIG. 10 is a diagram schematically showing an example of image data between a transmission terminal and a reception terminal (between a TV phone) and designation area information for designating an effective area of the image data in the image communication system of the present embodiment. is there. In FIG. 10, two videophones according to the present embodiment are prepared, that is, a transmission terminal and a reception terminal, and the state of image communication between them is shown.
[0086]
(A) of FIG. 10 shows an example in which designated area information is created by the area discriminating unit 18 and transmitted to the receiving terminal in the transmitting terminal that transmits image data. First, the designated area to be encoded by the transmitting terminal is a quadrangle having diagonal lines (hereinafter referred to as [(x0, y0), (x1, y1)]) connecting (x0, y0) and (x1, y1). The specific image data or still image data is written in the designated area. Next, in the designated area to be encoded, a designated area for inserting a moving image is set as [(x2, y2), (x3, y3)], and the moving image data input from the camera is fitted into the designated area.
[0087]
The transmitting terminal encodes the image data of the designated area [(x0, y0), (x1, y1)] generated as described above, and first designates the designated areas [(x0, y0), (x1, y1). ] And [(x2, y2), (x3, y3)] are transmitted, and then the encoded image data is transmitted.
[0088]
The receiving terminal receives the designated area information and the encoded image data transmitted from the transmitting terminal, decodes the image data, writes them in the memory, and recognizes the designated area of the moving image from the received designated area information. When only the moving image area is to be displayed on the monitor, moving image data is extracted from the designated area [(x2, y2), (x3, y3)] and displayed on the monitor. Of course, the designated area on the monitor when displayed on the monitor can be freely set on the receiving terminal side.
[0089]
Next, FIG. 10B shows an example in which the receiving terminal creates designated area information and transfers it to the transmitting terminal. First, a designated area [(x0, y0), (x1, y1)] to be encoded by the receiving terminal is set, and specific image data or still image data is written in the designated area. Next, in the designated area to be encoded, a designated area [(x2, y2), (x3, y3)] for inserting a moving image is set. Then, the designated area information of the designated areas [(x0, y0), (x1, y1)], [(x2, y2), (x3, y3)] is transmitted to the transmitting terminal side.
[0090]
The transmitting terminal receives the specified area information sent from the receiving terminal and writes it in the memory. Based on the specified area information, the transmitting terminal sets a specified area [(x0, y0), (x1, y1)] to be encoded, and writes specific image data or still image data in the specified area. Then, a designated area [(x2, y2), (x3, y3)] for fitting a moving picture is set within the designated area to be encoded, and the moving picture data input from the camera is fitted into the designated area. The transmitting terminal encodes the image data of the designated area [(x0, y0), (x1, y1)] generated in this way, and transmits only the encoded image data to the receiving terminal.
[0091]
The receiving terminal receives the encoded image data, decodes it, writes it in the memory, and recognizes the designated area of the moving image from the designated area information created by itself. When it is desired to display only the moving image area on the monitor, moving image data is extracted from the designated area [(x2, y2), (x3, y3)] and displayed on the monitor. Of course, the designated area on the monitor when displayed on the monitor can be freely set on the receiving terminal side.
[0092]
It should be noted that the moving image designation area as described above is managed by being written in the area designation management table 15 in the system control unit 14 shown in FIG.
[0093]
As described above, in the image communication system using the videophone of this embodiment, even if the image format at the time of encoding is fixed, the moving image is encoded with an arbitrary size smaller than the image size. It can be transmitted, and similarly received and displayed on a monitor.
[0094]
In the present embodiment, the description has been given by taking as an example a TV phone capable of processing audio information and image information as shown in FIG. 1, but the present invention is not limited to such a TV phone. Any device, method, and system for communicating moving images may be used.
[0095]
The present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or apparatus.
[0096]
【The invention's effect】
As described above, according to the present invention, even if the image format at the time of encoding is fixed, a moving image is inserted in units of the smallest image block in a size smaller than the image size, and in other regions A moving image and a still image can be combined and encoded as if a still image is embedded. Furthermore, by transmitting the encoded data and extracting only the moving image area on the receiving side and displaying it on the monitor, the moving image can be encoded and transmitted in an arbitrary size. Therefore, it is possible to change the image size according to the application and situation, and to execute the optimum encoding process, and to obtain a specific effect that the image communication can be performed with the optimum image quality.
[0097]
Further, since the difference value data of the still image will eventually be converged, the encoding target is concentrated in the moving image area, and the effect that the transfer efficiency of the moving image is improved is also obtained.
[0098]
In addition, by including a moving image area in the still image area, it is possible to transmit an image displayed by inserting a moving image in the still image, and the convenience for the operator is greatly improved.
[0099]
In addition, when displaying a moving image transmitted from the transmission side in a still image generated on the reception side, the position where the moving image is inserted is specified from the reception side, and the transmission side inserts the moving image at the specified position. Thus, there is no need to perform the fitting process and the moving image scaling process on the receiving side, so that the image communication system can be economically constructed and the image quality can be obtained in a good state.
[0100]
[Brief description of the drawings]
FIG. 1 is a block diagram of a videophone according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a detailed configuration of a transmission image editing unit in the present embodiment.
FIG. 3 is a block diagram showing a detailed configuration of a received image editing unit in the present embodiment.
FIG. 4 is a diagram for explaining a process for fitting data into a transmission image and a process for superimposing a reception image in the present embodiment.
FIG. 5 is a diagram for explaining a region designation method in the present embodiment.
FIG. 6 is a block diagram illustrating a detailed configuration of an image codec unit in the present embodiment.
FIG. 7 is a diagram illustrating an image format in the present embodiment.
FIG. 8 is a diagram showing a multiplexed frame configuration of image data in the present embodiment.
FIG. 9 is a diagram showing error correction frame synchronization in the present embodiment.
FIG. 10 is a diagram illustrating an example of a communication method in the present embodiment.
[Explanation of symbols]
1 microphone
2 Speaker
3 Voice processing part
4 Audio codec section
5 Camera
6 Monitor
7 Image processing section
8 Transmission image editor
9 Received image editing section
10 Specific image generator
11 Image codec section
12 Demultiplexer
13 Line interface section
14 System controller
15 Area specification management table
16 Specific image management table
17 Operation unit

Claims (13)

Area designating means for designating a first required area in which an image in one screen is transmitted and a desired second required area in the first required area as a moving image area;
Combination means for combining the moving image of the second required area and the still image of the first required area excluding the second required area based on the designation by the area designating means;
Encoding means for encoding the image data combined by the combination means;
And image data encoded by said encoding means, the first indicating the position of the region of the moving image of the area specifying unit required region area position information and the first of the designated first prescribed region by image communication apparatus characterized by a transmission means for transmitting the area position information of the two prescribed region to another image communication apparatus. And coded image data receiving means for receiving the area location information of the first predetermined region of the area position information and the second predetermined area,
Decoding means for decoding the image data received by the receiving means;
Based on said first area location information prescribed region and the region position information of the second predetermined area, for displaying process a moving image of the second predetermined area, the decoded image data by said decoding means 2. The image communication apparatus according to claim 1, further comprising extraction means for extracting a moving image from the image.   2. The image communication apparatus according to claim 1, wherein the encoding unit encodes image data in units of blocks, and the region specifying unit specifies regions in units of the blocks.   2. The image communication apparatus according to claim 1, wherein the combination means combines different moving images when at least two areas are designated by the area designation means.   The image communication apparatus according to claim 1, wherein the still image is a part of the moving image.   The image communication apparatus according to claim 1, wherein the encoding unit performs encoding based on an inter-frame difference. An audio processing unit for performing audio processing;
A voice encoding unit that encodes the voice data processed by the voice processing unit;
The image communication apparatus according to claim 2, further comprising voice transmission means for transmitting voice data encoded by the voice encoding unit. The first image communication apparatus
Area designating means for designating a first required area in which an image in one screen is transmitted and a desired second required area in the first required area as a moving image area;
Combination means for combining the moving image of the second required area and the still image of the first required area excluding the second required area based on the designation by the area designating means;
Encoding means for encoding the image data combined by the combination means;
And image data encoded by said encoding means, the first indicating the position of the region of the moving image of the area specifying unit required region area position information and the first of the designated first prescribed region by and a region position information of the second predetermined region and a transmission means for transmitting to another image communication apparatus,
The second image communication apparatus
And image data the encoded receiving means for receiving the area location information of the first predetermined region of the area position information and the second predetermined area,
Decoding means for decoding the image data received by the receiving means;
Based on said first area location information prescribed region and the region position information of the second predetermined area, for displaying process a moving image of the second predetermined area, the decoded image data by said decoding means An image communication system comprising: extraction means for extracting a moving image from the image.   9. The image communication system according to claim 8, wherein the encoding unit encodes image data in units of blocks, and the region specifying unit specifies regions in units of the blocks.   9. The image communication system according to claim 8, wherein the combination means combines different moving images when at least two areas are designated by the area designation means.   The image communication system according to claim 8, wherein the still image is a part of the moving image.   9. The image communication system according to claim 8, wherein the encoding means performs encoding based on an inter-frame difference. The first image communication apparatus includes:
A voice input unit that inputs voice data and performs voice processing;
A voice encoding unit that encodes the voice data input by the voice input unit;
Voice transmission means for transmitting the voice data encoded by the voice encoding unit;
The second image communication apparatus includes:
Voice receiving means for receiving voice data encoded by the voice encoding unit;
A voice decoding unit for decoding voice data received by the voice receiving means;
9. The image communication system according to claim 8, further comprising: an audio output unit that processes and outputs the audio data decoded by the audio decoding unit.

Images