JP2007281766A - Image transmitter and transmission method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide "an image transmitter and a transmission method thereof" which transmits an image with a decreased information amount of I frames at the start of image transmission so that a wait time of a receiver for receiving and displaying the image can be reduced. <P>SOLUTION: The image transmitter divides an image of a first frame into a plurality of sub images each comprising N×N pixels, generates a mosaic image P1 by filling pixel data of a prescribed pixel of a sub image to the whole pixels of the sub image and transmits the mosaic image P1 as an I frame. Then the image transmitter divides an image of a second frame into a plurality of smaller sub images each comprising n×n pixels (n<N), generates a mosaic image P2 by filling pixel data of a prescribed pixel of a sub image to the whole pixels of the sub image and transmits the mosaic image P2 as an I frame. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image transmission device and a transmission method thereof, and more particularly to an image transmission device and a transmission method thereof that transmit difference image data from an image transmitted last time.

  In recent years, mobile phones and personal digital assistants are equipped with cameras, and various image data transmission services such as an e-mail service with image data and a TV phone service can be used. However, in such a system, if data having a large amount of information such as image data is to be transmitted, the transmission band becomes a problem and the image data cannot be transmitted.

Therefore, when transmitting image data, an image encoding / decoding process is required, and a cellular phone or a portable information terminal currently employs an image encoding / decoding method called MPEG-4.
MPEG-4 is difference information between a frame as absolute value information (hereinafter referred to as an I frame) that makes sense independently as image data and a frame that does not make sense independently as image data. (Hereinafter referred to as P frame). The I frame is image data including all image information of one screen encoded as a still image, and the P frame is difference information from the image data of the previous frame and is not all image information of one screen.

FIG. 9 is a diagram schematically showing an operation example of the image data transmission / reception system using the MPEG-4. In the figure, for example, it is assumed that image data indicated by V1, V2, V3,..., VN,. First, the transmission transmission device CP1 encodes the image data V1 as a still image and transmits it as an I frame. On the other hand, the receiving device CP2 decodes and displays the one-screen data W1 based on the I frame.
Next, every time the image data V2, V3,... VN,... Are obtained, the transmitter CP1 obtains the image data V2, V3,... And the image data V1, V2,. And the difference information is encoded and transmitted as a P frame. On the other hand, each time the receiving device CP2 receives the P frame, the receiving device CP2 decodes and displays the image data by adding the data to the moving image data W1, W2,. However, the transmission apparatus CP1 appropriately transmits an I frame (for example, when transmitting image data VN) when the information amount of the difference information is large.
Thereby, for example, image data having an information amount of several Mbps can be transmitted as an information amount of several tens of kbps.

  By the way, at the start of image transmission, the transmission device CP1 must encode and transmit image data as an I frame. However, the I frame has a feature that the amount of information is larger than that of the P frame. Therefore, when the transmission device CP1 transmits an I frame to the reception device CP2 at the start of image transmission, if the information amount of the I frame is enormous, a waiting time until the reception device CP2 receives and displays the I frame. T was long, giving the user a sense of discomfort (see FIG. 10).

As a conventional technique, a plurality of encoding / decoding circuits and buffer memories are provided for a single frame memory in an image compression / decompression apparatus for transmission and reception, and image compression / decompression that shortens encoding / decoding processing by parallel processing There exists an apparatus (refer patent document 1).
Japanese Patent Laid-Open No. 7-147637

However, in the prior art, since the amount of information of the I frame transmitted at the start of image transmission is not reduced, the waiting time T until the receiving device CP2 receives and displays the I frame is still long.
Accordingly, an object of the present invention is to transmit an image with a reduced amount of I-frame information transmitted at the start of image transmission, and to reduce the waiting time for the reception device to receive and display an image, and the image transmission device Provide a transmission method.

  The present invention is an image transmission device that transmits difference image data from a previously transmitted image, and divides the image into a plurality of sub-images composed of N × N pixels, and pixel data of predetermined pixels of the sub-images An image processing unit that generates a mosaic image as pixel data of all pixels of an image and generates an original image or a difference image between the original image and the mosaic image after transmitting the mosaic image data; Transmits mosaic image data, and after transmitting the mosaic image data, transmits image data corresponding to the original image or a difference image between the original image and the mosaic image, and then transmits difference image data from the previously transmitted image. An image transmission unit.

  The image processing unit of the image transmission device converts the original image to n × n pixels (n) before the image transmission unit transmits image data corresponding to the original image or a difference image between the original image and the mosaic image. <N) is divided into a plurality of smaller sub-images, pixel data of predetermined pixels of the sub-image is generated as pixel data of all pixels of the sub-image, and the image transmission unit Generate data.

  The present invention is an image transmission method of an image transmission device that transmits difference image data from an image transmitted last time, and divides an image into a plurality of sub-images composed of N × N pixels, and pixels of predetermined pixels of the sub-images A step of generating a mosaic image by using data as pixel data of all pixels of the sub-image, a step of transmitting the mosaic image data first at the time of image transmission, an original image or the original image and the mosaic image after the transmission of the mosaic image data And a step of transmitting image data corresponding to the previously transmitted image.

  In the image transmission method, before transmitting image data corresponding to the original image or a difference image between the original image and the mosaic image, the image is converted into a plurality of smaller sub-pixels composed of n × n pixels (n <N). Dividing the image into images, and generating and transmitting a mosaic image using pixel data of predetermined pixels of the sub-image as pixel data of all pixels of the sub-image.

  According to the present invention, an image is divided into a plurality of sub-images composed of N × N pixels, and a mosaic image is generated using pixel data of predetermined pixels of the sub-image as pixel data of all pixels of the sub-image. At the time of transmission, the mosaic image data is transmitted first, and after the mosaic image data is transmitted, image data corresponding to the original image or a difference image between the original image and the mosaic image is transmitted. Since the difference image data is transmitted, it is possible to reduce the waiting time for the receiving apparatus to receive and display the image at the start of image transmission.

  According to the present invention, before transmitting image data corresponding to an original image or a difference image between the original image and the mosaic image, the image is converted into a plurality of smaller sub-images composed of n × n pixels (n <N). Since the mosaic image is generated and transmitted as the pixel data of all the pixels of the sub-image by dividing the pixel data of the predetermined pixel of the sub-image, the image can be displayed gradually and clearly. It is possible to display an image without giving a sense of incongruity.

(A) Outline of Image Transmission Processing According to the Present Invention FIG. 1 is a schematic diagram of image transmission processing according to the present invention. A transmission device (not shown) obtains an image of the first frame from an imaging device such as a camera when an instruction to start image transmission is given by the user, and the image is stored in a plurality of sub-pixels of N × N pixels (for example, 16 × 16 pixels). Divide into images. Then, the transmission apparatus uses the pixel data of predetermined pixels (for example, the upper left pixel) of the sub-image as pixel data of all the pixels of the sub-image, and generates a mosaic image P1. Thereafter, the transmission device encodes the mosaic image P1 as a still image and transmits it as an I frame. On the other hand, a receiving device (not shown) receives the I frame and decodes and displays the mosaic image P1 based on the I frame.
Next, the transmission apparatus acquires the second frame image from the imaging apparatus, and divides the image into a plurality of smaller sub-images of n × n pixels (n <N) (for example, 8 × 8 pixels). Then, the transmission apparatus uses the pixel data of predetermined pixels (for example, the upper left pixel) of the sub-image as pixel data of all the pixels of the sub-image, and generates a mosaic image P2. Thereafter, the transmission device encodes the mosaic image P2 as a still image and transmits it as an I frame. On the other hand, the receiving apparatus receives the I frame and decodes and displays the mosaic image P2 based on the I frame.
Next, the transmission device acquires the third frame image from the imaging device, encodes it as a still image, and transmits it as an I frame. On the other hand, the receiving apparatus receives the I frame and decodes and displays the image P3 based on the I frame.
After the image of the fourth frame, the transmitting device detects the difference between the image and the image one frame before, after obtaining the image, encodes this difference information, and transmits it as a P frame. On the other hand, the receiving device receives the P frame, adds the data to the image displayed one frame before, and decodes and displays the image. However, the transmitting apparatus can also transmit I frames as appropriate when the amount of difference information is large.
As a result, it is possible to shorten the waiting time for the receiving device to receive and display the image at the start of image transmission, and to display the image gradually and clearly without causing the user to feel uncomfortable. Can do.

  Next, when a wireless TV telephone communication is performed in a vehicle using an MPEG-4 image encoding / decoding method between transmission apparatuses of a mobile communication system, the receiving mobile phone receives and displays the image. An image transmission apparatus and a transmission method for reducing the waiting time will be described.

(B) Mechanism of wireless TV telephone communication FIG. 2 shows a case where wireless TV telephone communication is performed by multiplexing and transmitting image data, audio data, and computer data between the transmitting device CP1 and the receiving device CP2 of the mobile communication system. It is a figure which shows a structure.
The transmission device CP1 and the reception device CP2 are connected to each other by a physical layer, and perform data transmission in units of packets called MUX-PDUs. As shown in FIG. 2, the MUX-PDU has a flag and header arranged at the head and moving image data, audio data, and computer data alternately arranged behind the flag and header. Information representing the contents of the MUX-PDU is inserted into the header.
The AL / MUX layer is arranged above the physical layer, and the MUX-PDU is generated in the AL / MUX layer. Further, a higher layer is arranged above the AL / MUX layer, and a video channel, an audio channel, and a control / data channel are connected here. The higher layer is realized by an image codec, an audio codec, and a data communication unit, which will be described later.

FIG. 3 shows the data structure of the MPEG-4 stream. The MPEG-4 stream is inserted into the video channel shown in FIG. 2 and transmitted. In this MPEG-4 stream, as shown in FIG. 2, an I frame including all image elements of one screen is transmitted first, and thereafter a P frame consisting of difference information from the previous frame is sequentially transmitted at each frame timing. The
However, an I frame can be transmitted instead of the P frame.

(C) Configuration of Image Transmission / Reception Device FIG. 4 is a block diagram showing the configuration of the image transmission / reception device, which can be used as the transmission device CP1 and the reception device CP2 in FIG. The image transmission / reception apparatus includes a mobile terminal 1 and an in-vehicle device 2. The mobile terminal 1 includes a wireless unit 3, a baseband unit 4, an input / output unit 5, and a power supply unit 6, and the in-vehicle device 2 includes a CPU 41, a communication interface 42, a display unit 43, and an operation unit 44. Consists of

  In the figure, a radio frequency signal arriving from a base station via a radio channel for a mobile communication system is received by an antenna 11 and then input to a receiving circuit (RX) 13 via an antenna duplexer (DUP) 12. The The reception circuit 13 includes a high frequency amplifier, a frequency converter, and a demodulator. The radio signal is amplified with low noise by a low noise amplifier, and then mixed with a reception local transmission signal generated from a frequency synthesizer (SYN) 14 in a frequency converter, and converted into a reception intermediate frequency signal or a reception baseband signal. The output signal is digitally demodulated by a demodulator. As a demodulation method, for example, an orthogonal demodulation method corresponding to the QPSK method is used. The reception local transmission signal frequency generated from the frequency synthesizer 14 is instructed from the main control unit 21 provided in the baseband unit 4.

  The demodulated signal output from the demodulator is input to the baseband unit 4. The baseband unit 4 includes a main control unit 21, a demultiplexing unit 22, an audio codec 23, an image codec 24, an image processing unit 25, an LCD control unit 26, and a communication interface 27.

  The demodulated signal is identified by the main control unit 21 as control information or multimedia information such as image data. If it is multimedia information, it is input to the demultiplexing unit 22 where it is converted into audio data and image data. To be separated. Then, the audio data is input to the audio codec 23 where it is decoded and the reproduced audio signal is output from the speaker 34 of the input / output unit 5 as a loud voice. On the other hand, the image data is input to the image codec 24 and subjected to image decoding processing, and the image signal reproduced thereby is input to the CPU 41 of the in-vehicle device 2 connected via the communication interface 27. The CPU 41 inputs the image signal to the display unit 43 and displays it. The image signal can also be input to the LCD 32 via the LCD control unit 26 and displayed.

  On the other hand, the transmitted voice signal of the terminal user output from the microphone 33 of the input / output unit 5 is input to the audio codec 23 of the baseband unit 4, where it is encoded and then input to the demultiplexing unit 22. The The images output from the camera 31 are sequentially input to the image processing unit 25 of the baseband unit 4. The image processing unit 25 generates a mosaic image, which will be described later, for the first frame image and the second frame image when the image processing instruction is issued from the main control unit 21 when the TV telephone communication is started. In addition to being input to the codec 24, the images after the third frame are input to the image codec 24 as they are. The image codec 24 performs an encoding process on images sequentially input from the image processing unit 25. Specifically, I frame encoding processing is performed on the first and second mosaic images input from the image processing unit 25. Further, the control is performed so that the I frame encoding process is performed on the third frame image, and the P frame encoding process is performed on the fourth and subsequent frames. After the image encoding process is performed by the image codec 24, the image is input to the demultiplexing unit 22. In the demultiplexing unit 22, the encoded audio data and image data are multiplexed in a predetermined format, and the multiplexed transmission data is sent from the main control unit 21 to the transmission circuit (TX) 15 of the radio unit 3. Entered.

  The transmission circuit 15 includes a modulator, a frequency converter, and a transmission power amplifier. The transmission data is digitally modulated by a modulator, and then mixed with a transmission local oscillation signal generated from the frequency synthesizer 14 by a frequency converter and frequency-converted to a radio frequency signal. A QPSK system is used as the modulation system. The generated transmission radio frequency signal is amplified to a predetermined transmission level by a transmission power amplifier, then input to the antenna 11 via the antenna duplexer 12, and transmitted from the antenna 11 to the base station. .

5 and 6 are diagrams for explaining generation of a mosaic image for the first and second frame images.
After acquiring the first frame image from the camera 31, the image processing unit 25 divides the image into a plurality of sub-images of N × N pixels (for example, 16 × 16 pixels) (FIG. 5A). For example, each pixel of the sub-image 51 has pixel data as shown in FIG. Then, the image processing unit 25 sets pixel data of predetermined pixels (for example, the upper left pixel 52) of the sub image as pixel data of all the pixels of the sub image (FIG. 5C). The image processing unit 25 performs this process on all the sub-images to generate a mosaic image and inputs it to the image codec 24.
Next, the image processing unit 25 acquires the second frame image from the camera 31 and divides the image into a plurality of smaller sub-images of n × n pixels (n <N) (for example, 8 × 8 pixels). (FIG. 6 (A)). Each pixel of the sub-image 53 has pixel data as shown in FIG. Then, the image processing unit 25 sets pixel data of predetermined pixels (for example, the upper left pixel 54) of the sub image as pixel data of all the pixels of the sub image (FIG. 6C). The image processing unit 25 performs this process on all the sub-images to generate a mosaic image and inputs it to the image codec 24. On the other hand, in the third and subsequent frames, the image processing unit 25 acquires an image from the camera 31 and inputs it to the image codec 24 without performing any processing on the image.

  Although not shown, the power supply unit 6 includes a battery such as a lithium ion battery, a charging circuit for charging the battery, and a voltage generation circuit that generates a predetermined power supply voltage based on the output voltage of the battery. In addition, the input unit 5 displays information indicating the operation state of the terminal output from the main control unit 21, for example, an LCD 32 that displays a phone book, reception strength, battery remaining amount, and the like to start and end a call. And an illuminator 36 for illuminating the LCD 32 and the operation unit 35.

  Although not shown, the main control unit 21 includes a microprocessor and an internal memory including a ROM and a RAM, and includes a wireless channel connection control, a communication link establishment, and a normal control function. Further, the main control unit 21 receives that hands-free TV phone communication setting information performed by the operation unit 44 provided in the in-vehicle device 2 is input via the communication interface 27, and the vehicle is stopped by a vehicle speed sensor or the like (not shown). Is detected, the image data received from the transmission device CP1 is controlled to be displayed. When it is detected that the vehicle is traveling, the image data received from the transmission device CP1 is not displayed. To control.

(D) Operation of Image Transmission Processing Next, the operation of the image transmission / reception apparatus configured as described above will be described. FIG. 7 is a flowchart showing the operation contents of the transmission device CP1 and the reception device CP2.
Assume that the transmission device CP1 and the reception device CP2 operate the operation unit 35 of the portable terminal 1 or the operation unit 44 of the in-vehicle device 2 to start wireless TV telephone communication using MPEG-4. Accordingly, the camera 33 captures an image (step S1), and the main control unit 21 instructs the image processing unit 25 to generate a mosaic image for the first frame image. Based on this instruction, the image processing unit 25 acquires the first frame image from the camera 31 and then divides the image into a plurality of sub-images of 16 × 16 pixels (FIG. 5A). Is used as pixel data of all the pixels of the sub-image (FIG. 5C), a mosaic image is generated and input to the image codec 24 (step S2). The image codec 24 performs an I frame encoding process on the mosaic image, and transmits the I frame at the first frame timing (step S3). On the other hand, the receiving device CP2, which is the communication partner, receives the I frame transmitted from the transmitting device CP1, and the image codec 24 decodes the mosaic image P1 (FIG. 1) based on the I frame to be mounted on the vehicle. The information is displayed on the display unit 43 of the machine 2 (step S4).
Next, the main control unit 21 instructs the image processing unit 25 to generate a mosaic image for the second frame image. Based on this instruction, the image processing unit 25 acquires the image of the second frame from the camera 31, and then divides the image into a plurality of 8 × 8 pixel sub-images (FIG. 6A). Is used as the pixel data of all the pixels of the sub-image (FIG. 6C), a mosaic image is generated and input to the image codec 24 (step S5). The image codec 24 performs an I frame encoding process on the mosaic image and transmits the I frame (step S6). On the other hand, the receiving device CP2 receives the I frame transmitted from the transmitting device CP1, and the image codec 24 decodes the mosaic image P2 (FIG. 1) based on the I frame to display the vehicle-mounted device 2. This is displayed on the unit 43 (step S7).
The image processing unit 25 acquires the image of the third frame from the camera 31 and inputs it to the image codec 24 as it is. The image codec 24 performs an I frame encoding process on the third frame image input from the image processing unit 25, and transmits this I frame (step S8).
The receiving device CP2 receives the I frame transmitted from the transmitting device CP1, and the image codec 24 decodes the mosaic image P3 (FIG. 1) based on the I frame and displays it on the display unit 43 of the in-vehicle device 2. (Step S9). Next, the image processing unit 25 acquires the image of the fourth frame from the camera 31 and inputs it to the image codec 24 as it is. After obtaining the image from the image processing unit 25, the image codec 24 generates difference information between the image and the image one frame before (image of the third frame) (step S10), encodes the difference information, It is transmitted as a P frame (step S11). The receiving device CP2 receives the P frame transmitted from the transmitting device CP1, adds the data to the image (image P3) displayed one frame before, and decodes the image P4 (FIG. 1) to decode the in-vehicle device 2 It displays on the display part 43 (step S12). From the fifth frame onward, the transmission device CP1 and the reception device CP2 repeat the processes of steps S10 to S12. However, the transmitter CP1 can also transmit I frames as appropriate when the amount of difference information is large.
With the above processing, the receiving device CP2 can shorten the standby time for receiving and displaying the image, and can display the image gradually and clearly without causing the user to feel uncomfortable. it can.

(E) Modified Example of Image Transmission Processing In this embodiment, the transmission device CP1 performs I-frame encoding processing on the mosaic image P2 of the second frame and the original image P3 (FIG. 1) of the third frame for transmission. However, as shown in FIG. 8, the mosaic image P2 of the second frame and the original image P3 of the third frame (FIG. 1) can be transmitted after being subjected to the P frame encoding process.

  In the present embodiment, in the processing of steps S5 to S7, the transmission device CP1 generates and transmits a mosaic image P2 for the original image of the second frame, and the reception device CP2 displays this image. However, even if these processes are omitted, the receiving device CP2 can shorten the waiting time for receiving and displaying an image.

  In this embodiment, MPEG-4 is used as an image data encoding / decoding method. It can also be realized using other methods such as H.264.

  In the present embodiment, the case where wireless TV telephone communication is performed in a vehicle has been described. However, the present invention is also applied to a case where a vehicle is connected to a WWW server on the Internet via a network and image data is displayed from this server. The invention is applicable.

In addition, the present invention can be variously modified and changed within the scope of the gist of the present invention as to the configuration and control procedure of the image transmitting / receiving apparatus.

It is the schematic of the image transmission process which concerns on this invention. It is a figure which shows the mechanism at the time of performing wireless TV telephone communication. It is a figure which shows the data structure of an MPEG-4 stream. It is a block diagram which shows the structure of the image transmission / reception apparatus which concerns on this invention. It is a figure explaining producing | generating a mosaic image with respect to the image of the 1st frame. It is a figure explaining producing | generating a mosaic image with respect to the image of the 2nd frame. It is a flowchart which shows operation | movement of the image transmission / reception apparatus which concerns on this invention. It is a figure which shows the modification of the image transmission / reception operation | movement which concerns on this invention. It is the figure which represented typically the operation example of the transmission / reception system of the image data which uses MPEG-4. It is the figure which showed the display timing of the image which concerns on the conventional image transmission process.

Explanation of symbols

CP1 ... transmitting device CP2 ... receiving device P1, P2 ... mosaic image 1 ... mobile terminal 2 ... vehicle equipment 21 ... main control unit 24 ... image codec 25 ... image processing unit

Claims (4)

In the image transmission device that transmits the difference image data with the image transmitted last time,
The image is divided into a plurality of sub-images composed of N × N pixels, and pixel data of predetermined pixels of the sub-image is generated as pixel data of all pixels of the sub-image, and after transmitting the mosaic image data, An image processing unit for generating an original image or a difference image between the original image and the mosaic image;
At the time of image transmission, the mosaic image data is transmitted first, and after the mosaic image data transmission, image data corresponding to the original image or a difference image between the original image and the mosaic image is transmitted. An image transmission unit for transmitting difference image data from
An image transmitting apparatus comprising: The image processing unit, before the image transmission unit transmits image data corresponding to the original image or a difference image between the original image and the mosaic image, the original image from n × n pixels (n <N) A plurality of smaller sub-images, and generating a mosaic image using pixel data of predetermined pixels of the sub-image as pixel data of all pixels of the sub-image,
The image transmission unit transmits the mosaic image data;
The image transmission apparatus according to claim 1. In the image transmission method of the image transmission apparatus that transmits the difference image data with the image transmitted last time,
Dividing the image into a plurality of sub-images of N × N pixels, and generating a mosaic image using pixel data of predetermined pixels of the sub-image as pixel data of all pixels of the sub-image;
When transmitting an image, first transmitting the mosaic image data;
After transmitting the mosaic image data, transmitting the image data according to the original image or the difference image between the original image and the mosaic image, and thereafter transmitting the difference image data with the previously transmitted image;
An image transmission method for an image transmission apparatus, comprising: Before transmitting image data corresponding to the original image or a difference image between the original image and the mosaic image, the image is divided into a plurality of smaller sub-images composed of n × n pixels (n <N), Generating and transmitting a mosaic image using pixel data of predetermined pixels of the sub-image as pixel data of all pixels of the sub-image;
The image transmission method of the image transmission apparatus of Claim 3 characterized by the above-mentioned.

Images