JP4247666B2 - Image processing method and image processing system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing method and an image processing system, and in particular, image processing suitable for realizing a desired image quality and recording time by correcting a still image or a moving image captured by a digital camera or a mobile phone with a camera. Regarding technology.
[0002]
[Prior art]
Recently, digital cameras, camera-equipped mobile phones, and the like have become widespread, and are used by many people as one of the easy photographing methods. According to the mobile phone with a camera function disclosed in Patent Document 1, when a moving image is captured from the camera unit, basic image processing such as white balance is performed on the image data in the image processing unit, After the frame rate conversion unit adjusts the set frame rate value, the color number conversion unit adjusts the set number of colors. Thereafter, the size conversion unit adjusts the image data input from the color number conversion unit to the set size value, and the image compression unit compresses the image data at a set compression rate by an image compression method such as JPEG. When all these processes are completed, the CPU stores the image data in the RAM and displays the screen on the LCD.
[0003]
[Patent Document 1]
JP 2002-271671 A
[0004]
[Problems to be solved by the invention]
However, in the past, mobile terminals often cannot handle cases where the image size and format differ, and as a means of solving this, image processing is performed on the server side, and optimal processing is performed according to the specifications of the destination terminal. Means for applying are known. Such a method enables complicated processing by performing processing on the server, but has a drawback that the sender cannot confirm the processing result.
[0005]
The present invention has been made in view of such circumstances, and is implemented in a server in an image transmission system in which an image transmitted from a transmission side terminal is subjected to image processing by an image processing server and an image of a processing result is transmitted to the other side. It is an object of the present invention to provide an image processing method and system capable of confirming on the sender side in real time before sending the result of image processing to the other party or while sending the result to the other party.
[0019]
[Means for Solving the Problems]
  To achieve the purposeThe present inventionInIn such an image processing method, the transmission side terminal and the reception side terminal each have imaging means, and can transmit and receive compressed moving image data of the video captured by the imaging means of each terminal using a communication line. An image processing method applied to a communication system, the step of establishing a communication connection between the transmission side terminal and the reception side terminal and starting the delivery of compressed moving image data to each other; and Maintaining a communication connection between the connected image processing server and the transmitting terminal, transmitting compressed moving image data from the transmitting terminal to the image processing server, and receiving from the transmitting terminal The image processing server performs image processing on the compressed moving image data, and the processed compressed moving image data generated by the image processing is transmitted from the image processing server to the communication partner. A step of transmitting the processed compressed video data to the transmission side terminal using the communication connection maintained and the post-processing received from the image processing server. The compressed video data is restored, the image and the receptionTerminalSimultaneously displaying the image of the other party sent from the display device of the transmitting terminal, and displaying the moving image indicating the result after the processing and the image of the other party simultaneously on the display device. A step of receiving an instruction to change the image processing parameter and sending information corresponding to the instruction input from the operation unit of the transmission side terminal to the image processing server; and a moving image indicating the result after the processing. And the other party's image are simultaneously displayed on the display device, when an instruction to change the image processing parameter is input from the operation unit of the transmission side terminal, the image processing server according to the changed parameter The process of executing image processing and the compressed moving image data of the processing result generated by the image processing reflecting the changed parameter setting from the image processing server A step of transmitting the compressed moving image data from the image processing server to the transmitting side terminal by using the communication connection maintained and transmitting the compressed moving image data from the image processing server to the transmitting side terminal. The compressed moving image data is restored from the processing result after the parameter change received from the server, and the image and the reception are restored.TerminalAnd simultaneously displaying on the display device the image of the other party sent from.
[0020]
That is, the present invention is a technique that can be applied to image processing in the transmission of moving image between terminals having a so-called videophone function, and the result of image processing performed in the image processing server is a terminal on the receiver side (receiving terminal). Not only on the display device of the transmission source terminal (transmission side terminal). Thereby, communication can be performed while the sender himself / herself confirms the result of the real-time processing performed in the image processing server on the display device of the terminal at hand.
[0021]
When the processing result image returned from the image processing server is displayed on the display device of the transmitting terminal, the processing result image (self-captured image) and the image sent from the partner (captured image of the partner) Are preferably switched at the same time or as necessary. When simultaneously displaying the captured image of the user and the captured image of the other party returned from the image processing server, the size of the display area of each image and the screen division form (vertical division, left and right division, picture It is also possible to appropriately set (small window by in-picture).
[0023]
  In addition, the present inventionAccording to the aspect, it is possible to perform communication while changing the parameter setting as necessary and confirming the processing result in which the setting is reflected on the display device. The contents of the processing by the image processing server include, for example, making the face and body shape thinner / thick, changing the hairstyle (selecting a desired image from a plurality of compositing part images, and generating the original part image related to the selection) Addition of processing such as combining with an image increases entertainment.
[0027]
  An image processing system according to still another aspect of the present invention includes an image processing server connected to a communication line, a transmission-side terminal capable of transmitting / receiving compressed moving image data to / from each other via the image processing server, and reception An image processing system configured to perform image processing on the compressed moving image data received from the transmission side terminal via the communication line, and the image processing system. A connection maintenance control unit that maintains a communication connection with the transmission side terminal during image processing by the unit, and transmits the compressed moving image data after processing generated by the image processing unit to the reception side terminal, and Processing result transmission control means for transmitting the processed compressed video data to the transmission side terminal using the communication connection maintained by the connection maintenance control means. The transmission-side terminal includes an imaging unit that converts an optical image into an electrical signal, a communication unit that connects to the reception-side terminal and the image processing server via the communication line, and compression of a video imaged by the imaging unit. Video dataBeforeCommunication control means for transmitting to the image processing server and received from the image processing serverAfter the treatmentSignal processing means for restoring compressed moving image data, and restored by the signal processing meansAfter the treatmentVideoAnd the image of the other party sent from the receiving terminal side at the same timeDisplay means for displaying;Parameter changing operation means for performing an operation of changing an image processing parameter, wherein the image processing server performs image processing by the image processing means in accordance with the parameter changed by the parameter changing operation means, and the changed parameter The compressed moving image data generated by the image processing reflecting the setting of the transmission is transmitted to the receiving side terminal, and the processing result after the parameter change is shown using the communication connection maintained by the connection maintenance control means Control is performed to transmit the compressed moving image data to the transmission side terminal.
[0029]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0030]
[First Embodiment]
FIG. 1 is a configuration diagram of an image processing system according to the first embodiment of the present invention. As shown in the figure, this system includes a plurality of mobile phones 1 and 2 that function as communication terminals, and an image processing server 4 that relays communication between these terminals. 4 is connected via a network (communication line) 6. The network 6 is, for example, a wide area communication line network represented by the Internet, and includes various communication line networks such as a telephone line network, a mobile phone line network, and a dedicated line network regardless of wired or wireless.
[0031]
The mobile phone 1 is provided with a camera unit 12, an antenna 14, a display 16, an earpiece 17, a mouthpiece 18, and an operation unit 20. The operation unit 20 includes a dial button 21, a call button 22, a call disconnect / power button 23, a cross button 24, an execution button 25 and other various operation means. The operation unit 20 is not limited to a push-type switch member, and may be configured as a dial member, a lever switch, or the like.
[0032]
The display 16 is composed of a liquid crystal display capable of color display. Of course, other types of display devices such as organic EL may be applied. Although not explicitly shown in FIG. 1, the mobile phone 1 is provided with a speaker (denoted by reference numeral 27 in FIG. 2) for outputting sound during moving image reproduction and various sound effects, and sound during moving image shooting. A microphone (described as reference numeral 28 in FIG. 2) for capturing is provided.
[0033]
The camera unit 12 shown in FIG. 1 is fixed to a predetermined position of the casing 29 of the mobile phone 1, but a mode in which the camera unit can be pivoted so that the photographing direction can be freely changed is also possible. Moreover, a mode in which a plurality of camera units are mounted is also possible.
[0034]
FIG. 2 is a block diagram showing the configuration of the mobile phone 1. As shown in the figure, the cellular phone 1 is generally composed of a central processing unit (CPU) 30, a memory 32, a voice processing unit 34, a transmission / reception circuit 36, a speaker 37 of the earpiece 17, and a microphone 38 of the mouthpiece 18. It has the necessary components to realize the telephone function.
[0035]
The CPU 30 functions as a control unit that controls the operation of the entire mobile phone 1 according to a predetermined program, and also functions as an arithmetic device that performs various calculations. That is, the CPU 30 controls each circuit in the mobile phone 1 based on a command signal input from the operation unit 20, and performs communication control, photographing operation control, image processing control, display 16 display control, and memory 32 read / write control. And capacity management (including detection of free capacity).
[0036]
The CPU 30 is connected to the memory 32 via the bus 39. The memory 32 is a block including a non-volatile memory area in which a program executed by the CPU 30 and various data necessary for control are stored, and a volatile memory area used as a program development area and a calculation work area of the CPU 30 It is. The non-volatile memory area is used as a data storage area for a telephone directory and a call history, and also as a storage area for voice and images. Information on processing parameters in the image processing function of the present invention is stored in a nonvolatile memory area. The volatile memory area is also used as a temporary storage area for image data.
[0037]
The audio processing unit 34 is a means for processing or converting an audio signal in cooperation with the CPU 30. The audio processing unit 34 performs decoding processing of data received from the transmission / reception circuit 36 and encoding processing of data transmitted through the transmission / reception circuit 36, output signal generation processing to the speakers 27 and 37, and output from the microphones 28 and 38. Performs encoding processing of the input audio signal.
[0038]
When a communication connection with the communication partner is established in the call mode (mode using the telephone function), the voice input from the microphone 28 is converted into an electric signal and sent to the voice processing unit 34. The audio processing unit 34 converts the audio signal input from the microphone 38 into a digital signal (A / D conversion), performs necessary signal processing, and converts it into audio data of a predetermined format. The audio data (data for transmission) processed by the audio processing unit 34 is sent to the transmission / reception circuit 36 via the bus 39, converted into a predetermined transmission signal format, and then transmitted from the antenna 14.
[0039]
The signal received by the antenna 14 is demodulated by the transmission / reception circuit 36 and sent to the audio processing unit 34. The audio processing unit 34 converts the audio signal received from the transmission / reception circuit 36 into audio data in a predetermined format. The audio data (received data) generated by the audio processing unit 34 is converted into an analog audio signal (D / A conversion) and then sent to the speaker 37. Thus, the voice of the other party is audibly output from the speaker 37. Of course, the cellular phone 1 can transmit and receive various data such as image data, document data, and program data by using not only voice data but also an electronic mail function and an Internet browser function.
[0040]
Further, the cellular phone 1 includes a photographing lens 40 and a CCD image sensor (hereinafter referred to as CCD) 42, an analog processing unit 44 for processing an image signal obtained from the CCD 42, an A / D converter 45, an image. A processing unit 46 and a compression / decompression unit 47 are provided. In this example, a CCD is used as the image pickup element. However, the device is not limited to the CCD type, and other types of devices such as a CMOS type may be used.
[0041]
The light that has passed through the photographing lens 40 is focused on the light receiving surface of the CCD 42. A large number of photosensors (light receiving elements) are arranged in a plane on the light receiving surface of the CCD 42, and red (R), green (G), and blue (B) primary color filters are arranged in a predetermined arrangement corresponding to each photosensor. Arranged in structure. A color filter such as CMY can be used instead of the RGB color filter.
[0042]
The subject image formed on the light receiving surface of the CCD 42 is converted into a signal charge of an amount corresponding to the amount of incident light by each photosensor. The CCD 42 has an electronic shutter function that controls the charge accumulation time (shutter speed) of each photosensor according to the timing of the shutter gate pulse.
[0043]
The signal charges accumulated in each photosensor of the CCD 42 correspond to the signal charges on the basis of pulses (horizontal drive pulse φH, vertical drive pulse φV, overflow drain pulse) given from the timing generator (TG) 48 in accordance with instructions from the CPU 30. It is sequentially read out as a voltage signal (image signal). The image signal output from the CCD 42 is sent to the analog processing unit 44, and after necessary processing such as correlated double sampling (CDS) processing and gain adjustment, it is converted into a digital signal by the A / D converter 45. Is done. The digitized image data is sent to the image processing unit 46.
[0044]
The image processing unit 46 includes a synchronization circuit (a processing circuit that calculates the color of each point by interpolating a spatial shift of the color signal associated with the color filter array of the single CCD), a luminance / color difference signal generation circuit, and a gamma correction. The digital image signal processing means includes a circuit, a contour correction circuit, a white balance correction circuit, and the like, and processes an image signal while utilizing the memory 32 in accordance with a command from the CPU 30.
[0045]
The image processing unit 46 has been generated by performing predetermined processing such as white balance adjustment processing, gamma conversion processing, conversion processing to luminance signals (Y signals) and color difference signals (Cr, Cb signals) (YC processing). The image data is temporarily stored in the memory 32. Here, in the case where the image being shot is displayed on the display 16, the contents of the memory 32 are read out and sent to the display circuit 50, converted into a display signal format by the display circuit 50, and then supplied to the display 16. . As a result, the real-time video (through image) captured by the CCD 42 is displayed on the display 16, and the photographer can check the shooting angle of view (composition) from the display video on the display 16.
[0046]
When a photographing button on the operation unit 20 (for example, the execution button 25 is also used) is pressed, the CPU 30 detects this and performs a recording photographing operation. That is, exposure control of the CCD 42 and charge read-out control are performed in response to pressing of the photographing button. The image data for recording thus captured is subjected to YC processing and other predetermined signal processing in the image processing unit 46, and then sent to the compression / decompression unit 47, where it is compressed according to a predetermined compression format (for example, JPEG format). Is done. The compressed image data is stored in the image storage area of the memory 32.
[0047]
When the moving image shooting mode is selected, the recording operation starts when the shooting button is pressed, and then the moving image recording operation stops when the shooting button is pressed again. Note that the recording operation may be performed while the shooting button is continuously pressed, and the recording may be stopped by releasing the pressing.
[0048]
The sound at the time of moving image shooting is detected by the microphone 28, and the detection signal (audio signal) is converted into a predetermined signal format by the audio processing unit 34. The audio data is compressed together with the image data in the compression / decompression unit 47 according to a predetermined compression format (for example, MPEG format) and then recorded in the memory 32.
[0049]
At the time of image reproduction, image file data is read from the memory 32 in response to a user file selection operation or automatically. The compressed data read from the memory 32 is decompressed by the compression / decompression unit 47, converted into a display signal via the display circuit 50, and then output to the display. Thereby, the image content of the file is displayed on the display.
[0050]
The configuration of the mobile phone denoted by reference numeral 2 in FIG. 1 is the same as or similar to the configuration of the mobile phone 1 described above, and thus the description thereof is omitted.
[0051]
FIG. 3 is a functional block diagram showing the main part of the image processing server 4. The image processing server 4 includes a communication unit 52, a control unit 53, a memory 54, and an image processing unit 55. The communication unit 52 performs signal conversion and communication control for transmitting and receiving data according to a predetermined communication protocol (for example, TCP) with the mobile phones 1 and 2 connected via the network 6. That is, the communication unit 52 functions as an image data input unit and output unit.
[0052]
The control unit 53 controls the operation of the entire server including the communication unit 52. The memory 54 stores image data and other data under the control of the control unit 53. The image processing unit 55 is a processing unit including an image unit that performs transformation, replacement, synthesis, correction, and other conversion processing on an input image (processed image) in accordance with set processing parameters.
[0053]
As shown in FIG. 3, the data of the original image (processed image) 60 sent from the mobile phone 1 is input to the image processing server 4 via the communication unit 52, and the image processing unit via the control unit 53. 55.
[0054]
The image processing unit 55 performs image processing on the original image based on the image processing parameter specified by the control unit 53, and generates a processing result image (converted image) reflecting the parameter. The processing contents include various items such as brightness correction, contour emphasis correction, color correction, and pasting image synthesis, and adjustment values and the like are defined for each processing.
[0055]
The processing result image 62 generated by the image processing unit 55 is stored in the memory 54 in the image processing server 4 and is transmitted to the transmission source mobile phone 1 via the communication unit 52. The processing result image 62 sent out from the image processing server 4 is received by the mobile phone 1, and after necessary restoration processing is performed in the mobile phone 1, the image content is displayed on the display 16. The sender can confirm the processing result image 62 on the display 16 and can change the image processing parameters as necessary.
[0056]
Parameter information 63 input from the mobile phone 1 is input to the control unit 53 via the communication unit 52. The control unit 53 corrects the processing content of the image processing unit 55 based on the changed parameter. The processing result reflecting the parameter change is returned to the mobile phone 1 and displayed on the display 16 of the mobile phone 1.
[0057]
After confirming the processing result, if the image is OK, the mobile phone 1 sends a signal (transmission permission signal) indicating that to the image processing server 4. When the image processing server 4 receives the transmission permission signal from the cellular phone 1, it transmits the processing result image 62 in the memory 54 to the designated transmission destination in response to the transmission permission signal.
[0058]
Next, the operation of the image processing system configured as described above will be described.
[0059]
FIG. 4 is a flowchart showing the operation of this system. The mobile phone 1 on the sender side is connected to the mobile operator and transmits an image (in this case, a still image) to the image processing server 4 (step S110). The image transmission method may include an aspect in which an image is attached to an e-mail and the image is uploaded to the image processing server 4. Note that the image transmitted from the mobile phone 1 is not limited to the image captured by the camera unit 12 but may be an image received from the outside by e-mail or the like.
[0060]
Even after image transmission, the TCP connection between the mobile phone 1 of the transmission source and the image processing server 4 is maintained. The image processing server 4 receives the image sent from the mobile phone 1, and stores the image (original image) in the memory 54 (step S112). Thereafter, the image processing server 4 sends the received image to the image processing unit 55 and performs image processing (step S114). In the course of image processing, receiving terminal information is acquired as necessary, and image size adjustment (resizing) and the number of colors are determined in accordance with the specifications of the receiving terminal (step S116). The terminal information on the receiving side may be transmitted from the mobile phone 1 or 2, or may be registered in advance by a procedure such as user registration.
[0061]
Thus, when the required image processing is performed in the image processing server 4 and an image of the processing result is generated, the result image is stored in the memory 54 and is also stored in the image processing server using the maintained TCP connection. 4 is returned to the source mobile phone 1 (step S118). The mobile phone 1 receives the processing result image returned from the image processing server 4 (step S120), restores the image data, and displays the image data in a small size in the lower right corner of the display 16 (see step S122, FIG. 1). . FIG. 1 shows a state where a slightly dark original image is converted into a bright image by image processing.
[0062]
According to the screen example of FIG. 1, the processing result image is displayed in the small screen window 70 in the display 16, but the display method of the processing result is not limited to this example. There are various modes such as display screen switching, and a desired display method may be selected from a plurality of display methods.
[0063]
At this time, as shown in FIG. 1, a command menu 71 for “further processing” and a command menu 72 for “send OK” are displayed at the bottom of the screen of the display 16. These menu displays mean that the function “further processing button” is assigned to the operation button 73 and the function “send OK button” is assigned to the operation button 74.
[0064]
The sender confirms the display on the display 16 and presses the “send OK button” (operation button 74) if the displayed processing result is acceptable. The CPU 30 of the mobile phone 1 detects an input from the operation unit 20, and determines whether or not the processing result is OK (step S124 in FIG. 4). When the “send OK button” is pressed, the information is sent from the mobile phone 1 to the image processing server 4. The image processing server 4 receives the OK information (transmission permission) from the cellular phone 1 and completes the image processing (step S126), and transmits the processing result image stored in the memory 54 to the cellular phone 2 on the receiver side. (Step S140). The mobile phone 2 on the receiver side receives the image sent from the image processing server 4 (step S142), and displays the received image on the display (step S144).
[0065]
On the other hand, if the sender who has confirmed the processing result image displayed in step S122 is not satisfied with the processing result, the user presses the “further processing button” (operation button 73) to move to the parameter change input screen. Make adjustments.
[0066]
That is, when the “further process button” is pressed, a NO determination is made in step S124, and the process proceeds to step S130. In step S130, an operation for further adjusting the processing parameters based on the parameters used in the previous processing is accepted. For example, a step adjustment rate of 0, ± 1, ± 2 is defined for brightness correction and sharpness, and the user operates the +/− button (also used with the up and down keys of the cross button 24) as desired. Parameters can be entered. If the parameter value used in the previous process is “+1”, add “+1” to the value to correct it to “+2”, or adjust −1 from “+2”. It can be modified to “+1”.
[0067]
When parameter adjustment is performed in step S130, the information is sent to the image processing server 4. After acquiring the changed parameter information (step S132), the image processing server 4 returns to step S114 and performs image processing based on the changed parameter. At this time, the image processing server 4 uses the processed image stored in the memory 54 as basic data and adds processing according to the parameters set by the user. The converted image after processing is returned to the transmission source mobile phone 1 in step S118 and displayed on the display 16.
[0068]
By repeating this operation (steps S114 to S132), an image of an expression intended by the sender can be obtained. When an image that the sender seems to be happy with is obtained, the “send OK button” is pressed to end the process. Thus, the image processing server 4 transmits the processed image to the receiving-side mobile phone 2 that is the transmission destination in accordance with the selection of the sender.
[0069]
[Second Embodiment]
Next, a second embodiment of the present invention will be described.
[0070]
FIG. 5 is a flowchart showing the operation of the image processing system according to the second embodiment. Since the system configuration is similar to that of the first embodiment described above, description thereof is omitted.
[0071]
As shown in FIG. 5, the mobile phone 1 on the sender side connects to the mobile operator and transmits an image to the image processing server 4 according to a predetermined communication protocol (for example, TCP) (step S210). Even after image transmission, the TCP connection between the image processing server 4 and the mobile phone 1 is maintained. The image processing server 4 on the operator side receives the image (original image) sent from the mobile phone 1 and inputs it to the image processing device in the server (step S212). The image processing server 4 stores the input original image before processing in the memory 54 (step S214), and performs image processing on the input original image (step S216).
[0072]
When the processing result image is generated, the result image is stored in the memory 54 (step S218), and is returned from the image processing server 4 to the transmission source mobile phone 1 using the maintained TCP connection. (Step S220).
[0073]
The mobile phone 1 receives the processing result image returned from the image processing server 4 (step S222), restores the image data, and displays it on the display 16 (step S224).
[0074]
The sender confirms the display on the display 16 and presses the “send OK button” (operation button 74) if the displayed processing result is acceptable. On the other hand, if it is determined from the processing result that it is not desired to perform processing on this image, an operation of selecting a “do not process” command menu (not shown) is performed. The CPU 30 of the mobile phone 1 detects an input from the operation unit 20 and transmits a signal corresponding to the operation to the image processing server 4 (step S226).
[0075]
The image processing server 4 determines the signal received from the mobile phone 1 (step S228) and performs processing according to the instruction. That is, when the “transmission OK” command signal indicating that the processing result is adopted is received, the processing result image is read from the memory 54 (step S230) and transmitted to the transmission destination (step S234).
[0076]
On the other hand, if a “do not process” command signal indicating that the original image before processing is adopted is received in step S228, the original image before processing stored in the memory 54 is read (step S232). This is transmitted to the transmission destination (step S234). In this way, image transmission to the transmission destination is performed, and the process is terminated.
[0077]
[Third Embodiment]
Next, a third embodiment of the present invention will be described.
[0078]
FIG. 6 is a configuration diagram of an image processing system according to the third embodiment of the present invention. 6, parts that are the same as or similar to those in FIG. 1 are given the same reference numerals, and descriptions thereof are omitted. This system is a system (so-called videophone system) for performing communication while transmitting video and audio to and from each other between two camera-equipped mobile phones 1 and 2.
[0079]
Video captured by the camera unit 12 of the mobile phone 1 is sent to the mobile phone 2 on the other side via the image processing server 4 and displayed on the display 80 of the mobile phone 2. Similarly, an image taken by the camera unit 82 of the mobile phone 2 is displayed on the display 16 of the mobile phone 1.
[0080]
The image processing server 4 functions as a moving image processing apparatus that performs image processing of moving image data transmitted and received between the mobile phones 1 and 2. In a field such as a video phone, since real-time communication is required, it is necessary to quickly perform image processing such as correction processing for improving image quality. For this reason, the image processing server 4 includes a moving image processing apparatus capable of performing high-speed image processing on moving image data (referred to as compressed moving image data) digitally compressed by a compression technique such as MPEG in a compression space.
[0081]
MPEG, which is becoming a standard in the moving picture industry, and ITU-T H.264 used in videophones and the like. 261 and ITU-T H.264. 263, moving image data compressed by a compression method such as Nancy, which is adopted as a moving image compression standard for mobile phones, is an intra frame formed by data independent of the data of other frames, and data of other frames And an interframe formed by a difference with respect to. In such a case, if image quality correction is performed on an intra frame, an inter frame that depends on the intra frame is also corrected.
[0082]
For example, compressed moving image data in the MPEG format is composed of three types of frames: I frame, P frame, and B frame. The I frame is an intra frame, the P frame is an inter frame that depends only on the data of the previous frame, and the B frame is an inter frame that depends on both the frames before and after the I frame.
[0083]
FIG. 7 is a block diagram illustrating a configuration of the moving image processing apparatus 100 installed in the image processing server 4. As shown in the figure, the moving image processing apparatus 100 decodes the compressed moving image data G into an intra frame G 0 and an inter frame G 1, and decodes the intra frame G 0 obtained by the frame separating unit 102. A decoding unit 104 that obtains a decoded intra frame G01, a correction unit 106 that performs a correction process and other image processing on the decoded intra frame G01 to obtain a processed decoded intra frame G02, and encodes the processed decoded intra frame G02 An encoding unit 108 that obtains a processed intra frame G03, and a synthesizing unit 110 that synthesizes the processed intra frame G03 and the inter frame G1 to obtain a processed moving image G ′.
[0084]
With this configuration, the compressed moving image data is separated into an intra frame and an inter frame, and only the intra frame is decoded, image processed, and encoded to obtain a processed intra frame, and these processed intra frames are obtained. Then, the processed compressed moving image data is generated by combining the non-processing target interframe.
[0085]
Since only a part of the compressed moving image data is decoded, image processed, and encoded, the processing time can be shortened. Further, although the image processing is not directly performed on the inter frame, if the intra frame is corrected, the inter frame depending on the inter frame is indirectly processed.
[0086]
Note that the method of performing image processing in the MPEG space is not limited to the method described in FIG. For example, an intra frame (I frame) that does not depend on other frames for compressed video data, an inter frame (P frame) that does not depend on other frames, and an inter frame (P frame) that depends on other frames. B frame), and only the I frame and P frame are decoded, processed, and encoded to obtain processed frames, and the processed frames and B frames are combined. The compressed video data that has already been generated may be generated.
[0087]
In order to perform image processing, it is necessary to first decode the correction target portion by the decoding means. At this time, it is preferable to perform only decoding processing to the extent that image processing can be performed. The compression of moving image data is usually performed in stages. When decoding, moving picture data is decoded in stages by performing inverse operations at each stage in the reverse order of compression. For example, when MPEG compressed moving image data is decoded, the compressed moving image data is subjected to variable length decoding, the variable length decoded data is divided into motion vectors, DCT coefficients, etc., and then the inverse of motion compensated prediction and DCT conversion is performed. This requires a procedure such as IDCT conversion. On the other hand, since the DCT coefficient can be corrected in the image quality correction process, the decoding unit does not perform IDCT conversion, the DCT coefficient is supplied to the image processing unit (correction unit), and the correction unit processes the DCT coefficient. Can be applied. Further, when encoding later, the corrected DCT coefficient is used and there is no need to newly obtain a DCT coefficient, so that the encoding time can be shortened.
[0088]
Furthermore, there are the following methods. Image data compressed by MPEG includes a DCT coefficient and a motion vector for each frame in addition to header information indicating whether it is an intra frame or an inter frame. A motion vector is also required when decoding such compressed moving image data, or when encoding after performing processing such as image quality correction after decoding. Despite the fact that motion vectors do not significantly affect image quality such as gradation, the amount of computation for obtaining motion vectors during encoding is enormous, so decoding does not require new motion vectors when encoding. There is also an aspect in which the processing time is shortened by using the motion vector obtained at the time of encoding.
[0089]
Next, the operation of the image processing system according to the third embodiment configured as described above will be described. Here, the mobile phone 1 is described as the transmitting side, and the mobile phone 2 is described as the receiving side. However, in this system, the relationship between the transmitting side and the receiving side can be interchanged.
[0090]
FIG. 8 is a flowchart showing the operation of the system according to the third embodiment. First, a mobile phone 1 having a videophone function is used to connect to a communication partner and a videophone is started (step S310). The mobile phone 1 on the transmission side maintains a TCP connection with the image processing server 4. As described above, the image processing server 4 includes the moving image processing apparatus 100 capable of performing image processing at high speed in the MPEG space, and the compressed moving image data moving image transmitted from the transmitting-side mobile phone 1. The image is taken into the image processing apparatus (step S312).
[0091]
The image processing server 4 starts image processing (step S314), transmits the processed image to the mobile phone 2 (reception side terminal) as the communication partner, and is also maintained by the mobile phone 1 on the transmission side as the transmission source. The moving image of the processing result is transmitted using the TCP connection (step S316).
[0092]
The mobile phone 2 of the communication partner receives the moving image data sent from the image processing server 4 (step S318), restores it, and displays it on the display 80 (step S320).
[0093]
On the other hand, the transmitting-side mobile phone 1 receives the processing result sent from the image processing server 4 (step S322), restores it, and displays it on the screen of the display 16 (step S324, FIG. 6).
[0094]
According to the screen example shown in FIG. 6, the processed image is displayed in a small window 84 (for example, with a 1/6 size) at the lower right of the screen of the display 16. Thereby, the sender can confirm the processing result in real time, and can grasp what kind of image is currently sent to the other party.
[0095]
In the other display area (main display section) of the display 16, an image (moving image) of the other party sent from the mobile phone 2 of the other party of communication is displayed. The image being sent can be simultaneously confirmed on the display 16. The display method of the processing result is not limited to this example, and there are various modes such as splitting up and down, splitting left and right, switching display screens, and the like so that a desired display method can be selected from a plurality of display methods. May be.
[0096]
At this time, as shown in FIG. 6, a “further processing” command menu 71 is displayed at the lower left of the screen of the display 16. This menu display means that the function of “further processing button” is assigned to the operation button 73.
[0097]
If the sender who has confirmed the processing result displayed on the display 16 determines that he / she wants to change the processing contents, he / she presses the “further processing button” (operation button 73). When the “further processing button” is pressed, a sub menu 86 for setting parameter values of various parameters as shown in the figure is displayed on the display 16. The sender selects a desired item from the submenu 86 and adjusts the parameters.
[0098]
For example, for the processing items such as “brighten”, “sharp”, “suppress flicker”, “thinner”, “thicken”, etc., five levels of adjustment rates of 0, ± 1, ± 2 are defined respectively. The user can specify the parameter value by operating the +/− button (also used with the up and down keys of the cross button 24) according to his / her preference. For the item “change hairstyle”, a plurality of part images for pasting synthesis corresponding to various hair styles are prepared, and a desired type can be selected from them.
[0099]
The sender inputs an instruction from the operation unit 20 as necessary. The CPU 30 of the cellular phone 1 detects an input from the operation unit 20 and controls the process according to the determination of the input instruction content (step S326 in FIG. 8). When the “further processing button” is not pressed, it is determined that the current processing result is OK. Of course, a command menu (not shown) indicating that “this is OK” may be selected, and the determination may be made based on the selection of the menu.
[0100]
If it is determined in step S326 that the processing content is acceptable, the processing by the current processing parameter setting is continued in the image processing server 4 (step S328).
[0101]
On the other hand, if the processing result is not satisfactory, the sender presses the “further processing button” to set parameters (step S330). When parameter adjustment is performed in step S330, the information is sent to the image processing server 4. After acquiring the changed parameter information, the image processing server 4 returns to step S312 and performs image processing based on the changed parameter.
[0102]
The converted image after processing is transmitted to the mobile phone 2 of the communication partner and is returned to the mobile phone 1 of the transmission source and displayed on the display 16 (steps S314 to S324).
[0103]
By repeating this operation (steps S312 to S330), the sender can set parameter values until he / she likes his / her favorite image while immediately confirming the processed image on the display 16 at hand.
[0104]
If the processing content is determined, the call is maintained as it is (step S328). Thereafter, when the call is finished and the connection is disconnected (step S332), the process is finished (step S334).
[0105]
In each of the above-described embodiments, a mobile phone is exemplified as a communication terminal. However, the scope of application of the present invention is not limited to this, and the present invention can also be applied to various mobile terminals such as a PDA and a mobile personal computer.
[0108]
【The invention's effect】
  As explained aboveMain departureClearlyAccording to this, when sending and receiving video data between terminals that use the so-called videophone function, the image processed by the image processing server is displayed not only on the partner side (receiver side) but also on the sender terminal. Therefore, communication can be performed while the sender himself / herself confirms the processing result.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an image processing system according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing the configuration of the mobile phone shown in FIG.
3 is a functional block diagram showing the main part of the image processing server shown in FIG.
FIG. 4 is a flowchart showing the operation of the image processing system of this example.
FIG. 5 is a flowchart showing the operation of the image processing system according to the second embodiment of the present invention.
FIG. 6 is a configuration diagram of an image processing system according to a third embodiment of the present invention.
7 is a block diagram showing a configuration of a moving image processing apparatus installed in the image processing server shown in FIG. 6;
FIG. 8 is a flowchart showing the operation of the image processing system according to the third embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1, 2 ... Mobile phone, 4 ... Image processing server, 6 ... Network, 12 ... Camera part, 16 ... Display, 20 ... Operation part, 24 ... Cross button, 30 ... CPU, 32 ... Memory, 42 ... CCD, 52 ... Communication unit 53... Control unit 54 54 Memory 55 Image processing unit 80 Display 82 Camera unit

Claims (3)

The transmission-side terminal and the reception-side terminal each have an imaging unit, and are applied to a communication system that can transmit and receive compressed moving image data of video captured by the imaging unit of each terminal using a communication line. An image processing method comprising:
Establishing a communication connection between the transmitting terminal and the receiving terminal, and starting delivery of compressed video data to each other;
Maintaining a communication connection between the image processing server connected to the communication line and the transmitting terminal;
Sending compressed video data from the transmitting terminal to the image processing server;
Processing the compressed moving image data received from the transmission side terminal in the image processing server;
The compressed moving image data after processing generated by the image processing is transmitted from the image processing server to the receiving terminal as a communication partner, and the compressed moving image after processing is transmitted using the maintained communication connection. Transmitting image data to the transmitting terminal;
Restore the compressed moving image data after the processing received from the image processing server, and displaying simultaneously and the image and the receiving side mating sent from the terminal image in the display device of the transmission side terminal ,
When a moving image indicating the result after processing and the image on the other party are simultaneously displayed on the display device, an input of an instruction to further change image processing parameters is accepted and input from the operation unit of the transmitting terminal. Sending information corresponding to the received instruction to the image processing server;
When an instruction to change image processing parameters is input from the operation unit of the transmission side terminal when a moving image indicating the result after processing and the image on the other side are simultaneously displayed on the display device, the change is performed. Performing image processing in the image processing server in accordance with the parameters that have been performed;
The processing result compressed moving image data generated by the image processing reflecting the changed parameter setting is transmitted from the image processing server to the receiving terminal, and the maintained communication connection is used to transmit the compressed moving image data. A step of transmitting the compressed moving image data based on the processing result after the parameter change from the image processing server to the transmitting side terminal;
Restoring the compressed moving image data based on the processing result after the parameter change received from the image processing server, and simultaneously displaying the image and the counterpart image sent from the receiving terminal on the display device; ,
An image processing method comprising: An image processing system comprising: an image processing server connected to a communication line; and a transmission side terminal and a reception side terminal capable of transmitting / receiving compressed moving image data to / from each other via the image processing server,
The image processing server, image processing means for image processing the compressed moving image data received from the transmission side terminal via the communication line;
Connection maintenance control means for maintaining a communication connection with the transmission side terminal during image processing by the image processing means;
The compressed moving image data after processing generated by the image processing means is transmitted to the receiving side terminal, and the compressed moving image data after processing is transmitted using the communication connection maintained by the connection maintenance control means. Processing result transmission control means for transmitting to the transmission side terminal;
With
The transmitting terminal includes an imaging unit that converts an optical image into an electrical signal;
Communication means for connecting to the receiving terminal and the image processing server via the communication line;
Communication control means for transmitting compressed moving image data of the video imaged by the imaging means to the image processing server;
Signal processing means for restoring the compressed video data after the processing received from the image processing server;
Display means for simultaneously displaying the processed moving image restored by the signal processing means and the counterpart image sent from the receiving terminal ;
Provided with a parameter changing operation means for performing an operation of changing the image processing parameter,
The image processing server performs image processing by the image processing unit according to the parameter changed by the parameter change operation unit, and receives the compressed moving image data generated by image processing reflecting the changed parameter setting. And transmitting the compressed moving image data indicating the processing result after the parameter change to the transmitting terminal using the communication connection maintained by the connection maintaining control means. Image processing system. Imaging means for converting an optical image into an electrical signal;
A communication means for connecting to a communication partner terminal and an image processing server via a communication line;
Communication control means for transmitting compressed moving image data of the video imaged by the imaging means to the image processing server;
Connection maintenance control means for maintaining communication connection with the image processing server when image processing of the compressed moving image data is performed by the image processing server;
Signal processing means for restoring the compressed moving image data after image processing sent back from the image processing server and the compressed moving image data of the other party sent from the terminal of the communication partner;
Display means for simultaneously displaying the processing result image on the transmission side and the image on the other side restored by the signal processing means;
Parameter changing operation means for performing an operation of changing the image processing parameter,
When an instruction to change an image processing parameter is input by the parameter changing operation unit, image processing is performed by the image processing server in accordance with the changed parameter, and an image processing result reflecting the changed parameter setting is displayed. A communication terminal characterized in that a moving image to be displayed is displayed on the display means.

Images