JP2007228412A - Mobile terminal device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smoothly display a moving image by generating an interpolation frame while optimally controlling the quantity of information to be transmitted between terminals in a 3G-H.324M video telephone system configured by wireless connection. <P>SOLUTION: In addition to an H.223 control unit 1,000 or multiplexing/demultiplexing transmission data, an MPEG4/H.263 decoding unit 1,010, a decoded data storage unit 1,011, an MPEG4/H.263 encoding unit 1,060, an H.245 control unit 1,040 for analyzing H.245 control data and a parameter judgement unit 1,050, the mobile terminal device comprises an image interpolation unit 1,020 and an interpolated image storage unit 1,021, the quantity of information to be transmitted between terminals is controlled by the parameter judgement unit 1,050 and the H.245 control unit 1040, an optimal interpolation frame interval is determined by the parameter judgement unit 1,050, and interpolated image data corresponding to the interpolation frame interval are generated by the image interpolation unit 1,020. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to the third generation mobile communication standard 3G-H. The present invention relates to a moving image data processing technique in a videophone system using 324M, and more particularly to a portable terminal device that enables smooth moving image display by generating an interpolation frame while optimally controlling the amount of transmission information between terminals.

  Third generation mobile communication standard 3G-H. In 324M, after the call connection with the game terminal is completed, before the transmission of the moving image, the H.264 and H. H.245 protocol, transmission and reception capability information exchange (TerminalCapabilitySet), 223 Multiplex table entry information exchange (MultiplexEntrySend) and logical channel connection (OpenLogicalChannel). Further, even after a videophone call is started, transmission parameters can be changed by a mode request message (RequestMode). The moving image is H.264. Encoding is performed using H.263 or MPEG-4, and the encoded moving image data is H.264. The packet is input to the adaptation layer 223, divided into one or a plurality of variable length packets called video packet data units according to the designated multiplexing table entry, and a CRC code is added to each packet and transmitted.

  FIG. 13 shows 3G-H. It is a block diagram which shows the structure of the conventional portable terminal in the videophone system using 324M. In FIG. The H.223 control unit 1000 performs voice / video / H. H.245 performs multiplexing / demultiplexing of control data. 223 received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. It is transferred to the H.263 decoding unit 1010. MPEG4 / H. The data decoded by the H.263 decoding unit 1010 is stored in the decoded data storage unit 1011 and transferred to the display unit 1030 as decoded data 1120 to display an image.

  H. H.223 extracted by the H.223 control unit 1000. 245 control data 1141 is H.264. H. 245 is transferred to the control unit 1040 for parameter analysis. 245-related parameter information 1142 is notified to the parameter determination unit 1050, and parameters related to decoding are MPEG4 / H. H.263 decoding section 1010 is notified, and parameters related to encoding are MPEG4 / H. The H.263 encoding unit 1060 is notified. The imaging data 1171 captured by the imaging unit 1070 is MPEG4 / H. H.263 encoding unit 1060 and MPEG4 / H. The data encoded by the H.263 encoding unit 1060 is transmitted MPEG4 / H. H.263 encoded data 1101 223 is transferred to the control unit 1000.

  3G-H.COM configured by wireless connection. In the videophone system based on 324M, when the communication state of the transmission path deteriorates, the video packet data unit is lost, and the moving image display is delayed. When a part of the packet data unit is received by mistake due to noise on the transmission path or when it is completely lost, the video packet data unit is retransmitted by the video retransmission protocol.

However, images that have not been retransmitted within the time interval displayed on the receiving side are not decoded, and the display of moving images may be delayed. As a countermeasure, a technique has been developed in which moving image interpolation means is provided on the receiving side to interpolate frames lost during transmission. In order to improve the effect of moving image interpolation, moving image data is divided in time and sent to a plurality of transmission lines, and an image of a transmission line in which an error is mixed is used by using the image immediately before another transmission line. A technique for interpolation is disclosed (see, for example, Patent Document 1). Further, an interpolation processing technique that enables accurate reproduction of a missing frame on the receiving side by adding a plurality of motion vectors for a plurality of frames thinned out on the transmitting side to the transmission frame is disclosed (for example, Patent Document 2).
JP 2002-223437 A JP-A-10-271508

  As described above, the 3G-H. In a videophone system based on 324M, moving image interpolation is an effective means for countermeasures against loss of video packet data units that occur due to deterioration in the communication status of the transmission path. Furthermore, not only measures against loss of video packet data units due to the communication status of the transmission path, but also actively uses moving image interpolation to generate interpolated images between decoded images while suppressing the amount of transmission information Thus, it is conceivable to display a smooth moving image. At that time, it is necessary to consider the amount of processing and power consumption on the terminal side.

  The present invention relates to a 3G-H. An object of the present invention is to enable smooth video display by generating an interpolation frame while optimally controlling the amount of transmission information between terminals in a videophone system based on 324M.

  The present invention relates to a mobile terminal device that constitutes a videophone system configured by wireless connection, the means for determining an optimal interpolation frame interval according to the processing capability of the own station, and the interpolation frame interval from decoded image data And a means for generating interpolated image data according to the above. According to the above configuration, by generating interpolated image data from decoded image data at an optimal interpolation frame interval according to the processing capability of the local station, it is possible to reduce the amount of transmission information between terminals while enabling smooth moving image display. The amount of processing on the game side can be reduced.

  Further, in the present invention, the means for determining the interpolation frame interval determines an optimum interpolation frame interval according to the processing capability of the own station based on the reception frame rate of the decoded image data. According to the above configuration, even when the opposite side suppresses the transmission frame rate by reducing the transmission frame rate, the own interpolation side determines the optimum interpolation frame interval according to the processing capability of the own station based on the reception frame rate of the decoded image data. And a smooth moving image display can be performed.

  Furthermore, in the present invention, the means for determining the interpolation frame interval determines an optimum interpolation frame interval according to the processing capability of the own station based on the interpolation frame interval information designated by the user. According to the above configuration, the user uses the videophone system while judging the quality of the moving image display in order to determine the optimum interpolation frame interval according to the processing capability of the own station based on the interpolation frame interval information specified by the user. can do.

  Furthermore, in the present invention, there is provided means for notifying the other party of the reception frame rate capability set in the own station. According to the above configuration, it is possible to suppress the amount of transmission information between terminals according to the reception frame rate capability notified to the game side, and to reduce the processing amount on the game side.

  Furthermore, in the present invention, there is provided means for notifying a game of the presence / absence of means for generating the interpolated image data. According to the above configuration, the amount of transmission information between terminals can be suppressed according to the presence or absence of means for generating interpolated image data notified to the game side, and the processing amount on the game side can be reduced.

  Furthermore, in the present invention, there is provided means for notifying a game of an activity state of the means for generating the interpolated image data. According to the above configuration, the amount of transmission information between terminals can be suppressed in accordance with the presence / absence of means for generating interpolated image data notified to the game side and its activity state, and the processing amount on the game side can be reduced.

  Furthermore, in the present invention, there is provided means for notifying a player of a transmission frame rate request. According to the above configuration, it is possible to suppress the amount of transmission information between terminals in accordance with the transmission frame rate request notified to the game side, and to reduce the processing amount on the game side.

  According to the present invention, by generating interpolated image data from decoded image data at an optimal interpolation frame interval according to the processing capability of the terminal, the amount of transmission information between terminals can be suppressed while enabling smooth moving image display. In addition, the processing amount on the game side can be reduced, and the effect of optimizing the information processing amount and power consumption between terminals can be obtained.

  FIG. 1 shows 3G-H. It is a block diagram which shows the structure of the portable terminal which concerns on one Embodiment of this invention in the videophone system using 324M. In FIG. 1, as an addition to the configuration of the conventional portable terminal shown in FIG. 13, 1020 is an image interpolating unit that generates an interpolated image based on a frame rate specified from the decoded image, and 1021 is a decoded image and interpolated image. The decoded image and interpolated image storage unit 1080 stores video data stored in the video phone during videophone operation. Other components are the same as those of the conventional configuration shown in FIG.

  As data and signals in the configuration of FIG. 1, 1131 is decoded image data and interpolation image data transferred to the display unit 1030, 1151 is decoding time information, 1152 is interpolation frame interval information designated by the user, and 1153 is a game terminal Transmission frame interval information requested by the user, 1154 is the ON / OFF information of the image interpolation function designated by the user, 1121 is the image interpolation control including the interpolation frame generation number determined by the parameter determination unit and the interpolation processing ON / OFF information Information 1142 is the conventional H.264 format. H.245 including the transmission request frame interval, interpolation means presence / absence information, and interpolation ON / OFF information in addition to the H.245 control information. 245 control information 1141 is a conventional H.264 standard. H.245 including transmission request frame interval, interpolation means presence / absence information, and interpolation ON / OFF information in addition to the H.245 message. 245 messages, 1181 and 1102 are recorded data. Other data and signals are the same as those of the conventional configuration shown in FIG.

  FIG. 2 is a diagram showing decoding time information and decoded image management information stored in the decoded data storage unit 1011. 2000 is a correspondence table between decoded time information and decoded image data. 2001 is a previous decoding time t + 0, 2011. Indicates decoded image data at time t + 0, 2002 indicates the latest decoded time t + 3, and 2012 indicates decoded image data at time t + 3.

  FIG. 3 shows decoding time information, interpolation time information, decoded image data, and management information of the interpolated image data stored in the decoded image / interpolated image storage unit 1021. Reference numeral 3000 denotes the decoding time, the interpolation time, the decoded image data, and the interpolation. Correspondence table of image data, 3001 and 3011 and 3004 and 3014 are information related to decoding similar to the information of FIG. 2, 3002 and 3003 are interpolation times for generating an interpolation frame based on the frame rate notified by the image interpolation control information 1121, Reference numeral 3012 denotes interpolation image data corresponding to the interpolation time t + 1, and reference numeral 3013 denotes interpolation image data corresponding to the interpolation time t + 2.

  The 3G-H. Various embodiments of processing for generating interpolated image data for displaying a smooth moving image and optimizing the processing amount between terminals in a mobile terminal of a videophone system based on 324M will be described below.

(Embodiment 1)
FIG. 4 is a flowchart showing the procedure of the first embodiment in which the parameter determination unit 1050 determines the interpolation frame interval used for the interpolation process from the decoding time information so that the mobile terminal generates the interpolation image data. H. received from the game terminal. 223 multiplexed data is H.264. When demultiplexed by the H.223 control unit 1000, the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. It is input to the H.263 decoding unit 1010.

  MPEG4 / H. The H.263 decoding unit 1010 first transfers the new decoding time 2002 to the previous decoding time storage variable 2001 and transfers the new decoded image data 2012 to the previous decoding image data 2011. Next, the received MPEG4 / H. The decoding of the H.263 moving image encoded data 1111 is executed, the decoding time information acquired at this time is stored in the new decoding time 2002, and the decoded image data is stored in the new decoded image data 2012. Further, the new decoding time 2002 and the new decoded image data 2012 are transferred to the image interpolation unit 1020 as decoded data 1120.

  In the image interpolation unit 1020, when the decoded data 1120 is transferred, the new decoded time 3004 is transferred to the previous decoded time 3001, the new decoded image data 3014 is transferred to the previous decoded image data 3011, and then the newly transferred decoded data 1120 is transferred. The decoding time is stored in new decoding time 3004, and the decoded image data is stored in new decoded image data 3014.

  MPEG4 / H. H.263 decoding unit 1010 receives MPEG4 / H. The decoding time information acquired by decoding the H.263 moving image encoded data 1111 is notified to the parameter determination unit 1050 as the decoding time information 1151. When the decoding time information t1151 is notified to the parameter determination unit 1050, it is determined in the previous decoding time determination 4001 whether or not the previous decoding time storage variable a is an initial value. If a is an initial value, the previous decoding time storage processing is performed. At 4008, t is input to a, and at interpolation frame interval determination processing 4009, an initial value is input to the interpolation frame interval storage variable f.

  If it is determined in the previous decoding time determination 4001 that the previous decoding time storage variable a is not the initial value, t is input to b in the new decoding time storage processing 4002, and the previous frame is determined in the decoding time difference calculation processing 4003. And the decoding time difference c is compared with the minimum decoding time difference m in the minimum decoding time difference determination 4004. When the decoding time difference c is larger than the minimum decoding time difference m, a reciprocal of m is input to the frame rate variable f in the frame rate determination process 4005. If the decoding time difference c is smaller than the minimum decoding time difference m in the comparison result of the minimum decoding time difference determination 4004, the reciprocal of c is input to the frame rate variable f in the frame rate determination process 4006, and the minimum decoding time difference update process is performed. In 4007, c is input to the minimum decoding time difference m.

  The parameter determination unit 1050 notifies the image interpolation unit 1020 of the interpolation frame interval f determined as described above as the image interpolation control information 1121. When the image interpolation control information 1121 is notified, the image interpolation unit 1020 calculates the interpolation times t + 1 and t + 2 from the previous decoding time 3001, the new decoding time 3004, and the image interpolation control information 1121, and stores them at the interpolation times 3002 and 3003, respectively. Also, the interpolated image data Pt + 1 at the interpolation time t + 1 and the interpolated image data Pt + 2 at the interpolated time t + 1 are calculated using the pre-decoded image data 3011 and the new decoded image data 3014, and stored in the interpolated image data 3012 and 3013, respectively. When the interpolation processing is completed in the image interpolation unit 1020 and the decoded data and the interpolation data are stored in the interpolation image storage unit 1021, the decoded data and the interpolation image data 1131 are transferred to the display unit 1030 to display an image. .

(Embodiment 2)
FIG. 5 is a flowchart showing the procedure of the second embodiment in which the parameter determination unit 1050 determines the interpolation frame interval used for the interpolation process from the interpolation frame interval specified by the user in order for the mobile terminal to generate the interpolation image data. . H. received from the game terminal. 223 multiplexed data is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment.

  When the user specifies an interpolation frame interval for the receiving terminal, the parameter determination unit 1050 is notified as user-specified interpolation frame interval information 1152. When the user-specified interpolation frame interval fr1152 is notified, the parameter determination unit 1050 determines whether or not the user-specified interpolation frame interval fr is within a specifiable range in the interpolation frame interval range determination 5001, and fr can be specified. If it is smaller than the lower limit value min of this range, min is input to the interpolation frame interval f in the interpolation frame interval determination processing 5002. When the determination result of the interpolation frame interval range determination 5001 indicates that fr is larger than the upper limit value max of the specifiable range, the interpolation frame interval determination processing 5004 inputs max to the interpolation frame interval f. When the determination result of the interpolation frame interval range determination 5001 indicates that fr can be specified, the interpolation frame interval determination processing 5003 inputs the user-specified interpolation frame interval fr as the interpolation frame interval f. The parameter determination unit 1050 notifies the image interpolation unit 1020 of the interpolation frame interval f determined as described above as the image interpolation control information 1121.

  After the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates the interpolation times t + 1 and t + 2, and further calculates the interpolated image data Pt + 1 and the interpolated image data Pt + 2 by interpolation processing, which are decoded. Processing until image data and interpolated image data 1131 are transferred to the display unit 1030 and an image is displayed is the same as in the first embodiment.

(Embodiment 3)
FIG. 6 is a flowchart showing the procedure of the third embodiment for the purpose of controlling the transmission frame interval of the moving image of the game terminal and reducing the processing amount of the game terminal by using the image interpolation means by the mobile terminal. FIG. When the user specifies the interpolation frame interval fr to the receiving terminal, the parameter determination unit 1050 is notified as user-specified interpolation frame interval information fr1152. The processing from when the interpolation frame interval information fr is notified until the parameter determination unit 1050 determines the interpolation frame interval f is the same as in the second embodiment.

  The parameter determination unit 1050 notifies the image interpolation unit 1020 of the interpolation frame interval f determined as described above as the image interpolation control information 1121. Further, parameter determination section 1050 determines fb satisfying fb ≦ f in own station reception frame rate processing capability determination processing 5005, and inputs fb to own station reception frame rate processing capability fa. The parameter determination unit 1050 sets the received frame rate processing capability fa determined in this way to H.264. H.245 related parameter information 1142 245 control unit 1040 is notified.

  After the call connection is completed, The H.245 control unit 1040 sends an H.264 message to the capability information exchange message (TerminalCapabilitySet) transmitted to the game terminal. The reception frame rate processing capability of the local station notified by the H.245 related parameter information 1142 is set. H.245 message 1141 223 control unit 1000 is notified. H. H.223 multiplexed by the H.223 control unit 1000. 245 message is received at the game terminal, and the H.264 message of the game terminal is received. The H.223 control unit performs demultiplexing processing. H.245 message 1141 245 control unit 1040 is notified. H. The H.245 control unit 1040 analyzes the capability information exchange message (TerminalCapabilitySet), and the setting parameters including the reception frame rate processing capability of the game terminal are H.264. The parameter determination unit 1050 is notified as the H.245 related parameter information 1142. The parameter determination unit 1050 uses the MPEG-4 / H. 263 encoding unit 1060 is notified.

  MPEG4 / H. The H.263 encoding unit 1060 uses the received frame rate processing capability of the game terminal notified as the encoding-related parameter 1161 as the maximum frame rate, and converts the captured image data 1171 transferred from the imaging unit 1070 to MPEG4 / H. H.263 encoding unit 1060, and the transmission MPEG4 / H. H.263 encoded data 1101 223 to the control unit 1000. This encoded data is H.264. The data is multiplexed by the H.223 control unit 1000 and transmitted to the game terminal. H. received from the game terminal. 223 multiplexed data is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment. Further, after the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates interpolation times t + 1 and t + 2, and further calculates interpolation image data Pt + 1 and interpolation image data Pt + 2 by interpolation processing. Is transferred to the display unit 1030 as decoded image data and interpolated image data 1131, and the processing until the image is displayed is the same as in the first embodiment.

(Embodiment 4)
FIG. 7 is intended to reduce the processing amount of the game terminal by controlling the transmission frame interval of the moving image by notifying the game terminal that the own station terminal has the image interpolation means. FIG. 10 is a flowchart showing a procedure of the fourth embodiment. When the user specifies the interpolation frame interval fr to the receiving terminal, the parameter determination unit 1050 is notified as user-specified interpolation frame interval information fr1152. The processing from when the interpolation frame interval information fr is notified until the parameter determination unit 1050 determines the interpolation frame interval f is the same as in the second embodiment.

  The parameter determination unit 1050 notifies the image interpolation unit 1020 of the interpolation frame interval f determined as described above as the image interpolation control information 1121. Further, the parameter determining unit 1050 inputs the information on the image interpolation means availability information e in the interpolation means availability information setting processing 5006, and sets the set image interpolation means availability information e to H.264. H.245 related parameter information 1142 245 control unit 1040 is notified.

  After the call connection is completed, The H.245 control unit 1040 sends an H.264 message to the capability information exchange message (TerminalCapabilitySet) transmitted to the game terminal. H.245-related parameter information 1142 is set to the image interpolation means availability information of the local station terminal. H.245 message 1141 223 control unit 1000 is notified. H. H.223 multiplexed by the H.223 control unit 1000. 245 message is received at the game terminal, and the H.264 message of the game terminal is received. The H.223 control unit performs demultiplexing processing. H.245 message 1141 245 control unit 1040 is notified. H. The H.245 control unit 1040 analyzes the capability information exchange message (TerminalCapabilitySet), and the setting parameters including the reception frame rate processing capability of the game terminal and the information on the image interpolation means are set to H.264. The parameter determination unit 1050 is notified as the H.245 related parameter information 1142. The parameter determination unit 1050 changes the reception frame rate processing capability of the game terminal according to the image interpolation means availability information of the game terminal.

  FIG. 8 is a flowchart of processing in which the parameter determination unit 1050 changes the reception frame rate processing capability of the game terminal. In FIG. 8, when the reception frame rate processing capability fo of the game terminal and the image interpolation means availability information eo of the game terminal are notified, the interpolation means availability information determination 5101 determines the image interpolation means availability information eo of the game terminal. If eo is “present”, fa is set to fa ≦ fo in the received frame rate processing capability change setting 5102, and fa is input to the received frame rate processing capability f. If eo is “not present”, reception is performed. In the frame rate processing capability setting 5103, fo is input to the reception frame rate processing capability f.

  The parameter determination unit 1050 includes the received frame rate processing capability f determined as described above in the encoding-related parameter 1161, and the MPEG4 / H. 263 encoding unit 1060 is notified. When the encoding related parameter 1161 is notified, MPEG4 / H. The H.263 encoding unit 1060 encodes the imaging data 1171 notified from the imaging unit 1070 using the notified reception frame rate processing capability f of the player terminal as the maximum frame rate, and transmits MPEG4 / H.264. H.263 encoded data 1101 223 to the control unit 1000. This encoded data is H.264. The data is multiplexed by the H.223 control unit 1000 and transmitted to the game terminal.

  H. received from the game terminal. 223 multiplexed data is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment. Further, after the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates interpolation times t + 1 and t + 2, and further calculates interpolation image data Pt + 1 and interpolation image data Pt + 2 by interpolation processing. Is transferred to the display unit 1030 as decoded image data and interpolated image data 1131, and the processing until the image is displayed is the same as in the first embodiment.

(Embodiment 5)
FIG. 9 shows that the local terminal has image interpolation means and notifies the game terminal of the state (ON / OFF) of the image interpolation means, so that the game terminal controls the transmission frame interval of the moving image. It is a flowchart which shows the procedure of Embodiment 5 aiming at reducing a processing amount. When the user designates the interpolation frame interval fr and the image interpolation means state s (ON / OFF) to the receiving terminal, the parameter determination unit 1050 is notified as user-specified interpolation frame interval information fr1152 and interpolation means state information s1154. . The processing until the parameter determination unit 1050 determines the interpolation frame interval f after the interpolation frame interval information fr and the interpolation means state information s are notified is the same as that of the second embodiment.

  Further, the parameter determination unit 1050 inputs the information on the image interpolation means availability information e in the interpolation means availability information setting processing 5006, determines the image interpolation means availability information e in the interpolation means availability information determination 5007, and e is “ In the case of “comprising”, the interpolation means state information s is input to the interpolation means state information st in the interpolation means state setting 5008. The parameter determination unit 1050 notifies the image interpolation unit 1020 as the image interpolation control information 1121 of the interpolation frame interval f determined as described above and the set interpolation means state information st. Further, the parameter determination unit 1050 converts the image interpolation means provision information e and the interpolation means status information st into H.264. H.245 related parameter information 1142 245 control unit 1040 is notified.

  After the call connection is completed, The H.245 control unit 1040 sends an H.264 message to the capability information exchange message (TerminalCapabilitySet) transmitted to the game terminal. H.245-related parameter information 1142 is set to the image interpolation means availability information and interpolation means status information of the local station. H.245 message 1141 223 control unit 1000 is notified. H. H.223 multiplexed by the H.223 control unit 1000. 245 message is received at the game terminal, and the H.264 message of the game terminal is received. The H.223 control unit performs demultiplexing processing. H.245 message 1141 245 control unit 1040 is notified. H. The H.245 control unit 1040 analyzes the capability information exchange message (TerminalCapabilitySet), and the setting parameters including the reception frame rate processing capability of the terminal, the image interpolation means availability information, and the interpolation means status information are H.264. The parameter determination unit 1050 is notified as the H.245 related parameter information 1142. The parameter determination unit 1050 changes the reception frame rate processing capability of the game terminal according to the image interpolation means availability information and the interpolation means status information of the game terminal.

  FIG. 10 is a flowchart of processing in which the parameter determination unit 1050 changes the reception frame rate processing capability of the game terminal. In FIG. 10, when the reception frame rate processing capability fo of the game terminal, the image interpolation means availability information eo of the game terminal, and the image interpolation means status information sto of the game terminal are notified, the interpolation means availability information determination 5101 indicates the The image interpolation means provided information eo is determined. When eo is “comprising”, the interpolation means state information determination 5104 determines the image interpolation means information sto of the game terminal. When sto is “ON”, fa that satisfies fa ≦ fo is determined in the received frame rate processing capability change setting 5102, fa is input to the received frame rate processing capability f, and when eo is “not present” or sto Is “OFF”, fo is input to the reception frame rate processing capability f in the reception frame rate processing capability setting 5103.

  The parameter determination unit 1050 includes the received frame rate processing capability f determined as described above in the encoding-related parameter 1161, and the MPEG4 / H. 263 encoding unit 1060 is notified. When the encoding related parameter 1161 is notified, MPEG4 / H. The H.263 encoding unit 1060 encodes the imaging data 1171 notified from the imaging unit 1070 using the notified reception frame rate processing capability f of the player terminal as the maximum frame rate, and transmits MPEG4 / H.264. H.263 encoded data 1101 223 to the control unit 1000. This encoded data is H.264. The data is multiplexed by the H.223 control unit 1000 and transmitted to the game terminal. H. received from the game terminal. 223 multiplexed data is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment. Further, after the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates interpolation times t + 1 and t + 2, and further calculates interpolation image data Pt + 1 and interpolation image data Pt + 2 by interpolation processing. Is transferred to the display unit 1030 as decoded image data and interpolated image data 1131, and the processing until the image is displayed is the same as in the first embodiment.

(Embodiment 6)
FIG. 11 shows that when the local station terminal has an image interpolation means and performs videophone recording during the videophone operation, the transmission frame interval of the moving image of the game terminal is controlled to reduce the processing amount of the game terminal. It is a flowchart which shows the procedure of Embodiment 6 aiming at this. When the user specifies the interpolated frame interval fr, the requested transmission frame interval fd smaller than the fr requested by the player terminal, and the state s (ON / OFF) of the image interpolating means during the videophone operation, the user-specified interpolated frame interval information The parameter determination unit 1050 is notified of fr 1152, requested transmission frame interval information fd 1153, and interpolation means state information s. The processing until the parameter determination unit 1050 determines the interpolation frame interval f after the interpolation frame interval information fr, the requested transmission frame interval information fd, and the interpolation means state information s are notified is the same as that of the second embodiment.

  Further, the parameter determining unit 1050 inputs the information on the image interpolation means availability information e in the interpolation means availability information setting processing 5006, determines the image interpolation means availability information e in the interpolation means availability information determination 5007, and e In the case of “comprising”, the interpolation means state information s is determined by the interpolation means state determination 5009, and when s is “ON”, the interpolation means state information st is added to the interpolation means state information st by the interpolation means state setting 5008. Enter. Next, in request transmission frame interval range determination 5010, it is determined whether or not the request transmission frame interval fd is within a requestable range. When fd is smaller than the lower limit value fqmin of the specifiable range, fqmin is input to the request transmission frame interval fq in the request transmission frame interval determination processing 5011. When fd is larger than the upper limit value fqmax of the specifiable range, request transmission is performed. In the frame interval determination processing 5013, fqmax is input to the request transmission frame interval fq. When fd is within a specifiable range, the request transmission frame interval fq is specified as the request transmission frame interval fq in the request transmission frame interval determination processing 5012. Enter.

  The parameter determination unit 1050 notifies the image interpolation unit 1020 as the image interpolation control information 1121 of the interpolation frame interval f determined as described above and the set interpolation means state information st. Further, the parameter determination unit 1050 sets the request transmission frame interval fq to H.264. H.245 related parameter information 1142 245 control unit 1040 is notified. After the call connection is completed, The H.245 control unit 1040 sends an H.264 message to the mode request message (RequestMode) transmitted to the game terminal. The request transmission frame interval information notified in the H.245 related parameter information 1142 is set. H.245 message 1141 223 control unit 1000 is notified. H. H.223 multiplexed by the H.223 control unit 1000. 245 message is received at the game terminal, and the H.264 message of the game terminal is received. The H.223 control unit performs demultiplexing processing. H.245 message 1141 245 control unit 1040 is notified. H. The H.245 control unit 1040 analyzes the mode request message (RequestMode), and setting parameters including the requested transmission frame rate are H.264. The parameter determination unit 1050 is notified as the H.245 related parameter information 1142. The parameter determination unit 1050 includes the MPEG-related parameters 1161 in the encoding related parameters 1161. 263 encoding unit 1060 is notified.

  When the encoding related parameter 1161 is notified, MPEG4 / H. The H.263 encoding unit 1060 encodes the imaging data 1171 notified from the imaging unit 1070 using the notified request transmission frame rate as the maximum frame rate, and transmits MPEG4 / H. H.263 encoded data 1101 223 to the control unit 1000. This encoded data is H.264. The data is multiplexed by the H.223 control unit 1000 and transmitted to the game terminal. H. received from the game terminal. 223 multiplexed data is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment.

  Further, after the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates interpolation times t + 1 and t + 2, and further calculates interpolation image data Pt + 1 and interpolation image data Pt + 2 by interpolation processing. Is transferred to the display unit 1030 as decoded image data and interpolated image data 1131, and the processing until the image is displayed is the same as in the first embodiment. In addition, when recording a videophone while the videophone is in operation, 223 received from the terminal 223 received by the control unit 1000. The H.223 multiplexed data is used as recorded data 1181 as it is. It is stored in the recorded data storage unit 1080 together with the H.223 multiplexing table.

  FIG. 12 is a flowchart for determining the interpolation frame interval used for the interpolation processing from the interpolation frame interval specified by the user in the parameter determination unit 1050 when the videophone recorded in the recording data storage unit 1080 is reproduced. When the user specifies the interpolation frame interval fr and the state s (ON / OFF) of the image interpolation means, the parameter determination unit 1050 is notified as user-specified interpolation frame interval information fr1152 and interpolation means state information s1154. The processing until the parameter determination unit 1050 determines the interpolation frame interval f after the interpolation frame interval information fr and the interpolation means state information s are notified is the same as that of the second embodiment.

  Further, the parameter determination unit 1050 inputs the interpolation unit state information s to the interpolation unit state information st in the interpolation unit state setting 5008. The parameter determination unit 1050 notifies the image interpolation unit 1020 as the image interpolation control information 1121 of the interpolation frame interval f determined as described above and the set interpolation means state information st. When playback of the recorded videophone is started, the recorded data storage unit 1080 stores the H.264 data. The recorded data 1102 and the multiplexing table are transferred to the H.223 control unit 1000. The recorded data 1102 is H.264. 223 controller 1000 demultiplexes the received MPEG4 / H. H.263 moving image encoded data 1111 is MPEG4 / H.264. The processing from the input to the H.263 decoding unit 1010 until the new decoding time 3004 and the new decoded image data 3014 are stored in the image interpolation unit 1020 is the same as that in the first embodiment. Further, after the image interpolation control information 1121 is notified to the image interpolation unit 1020, the image interpolation unit 1020 calculates interpolation times t + 1 and t + 2, and further calculates interpolation image data Pt + 1 and interpolation image data Pt + 2 by interpolation processing. Is transferred to the display unit 1030 as decoded image data and interpolated image data 1131, and the processing until the image is displayed is the same as in the first embodiment.

  The mobile terminal device according to the present invention generates interpolated image data from decoded image data at an optimal interpolated frame interval according to the processing capability of the terminal, thereby enabling smooth moving image display and an amount of transmission information between terminals. The amount of processing on the game side can be reduced, and the amount of information processing and power consumption can be optimized between terminals. Generation of interpolation frames while optimally controlling the amount of transmission information between terminals Therefore, it is useful as a portable terminal device or the like that enables smooth moving image display.

The block diagram which shows the structure of the portable terminal device which concerns on one Embodiment of this invention. The correspondence table of the decoding time and decoding image data in a decoding data storage part. The correspondence table of the decoding time and interpolation time, and decoding image data and interpolation image data in a decoding image and interpolation image storage part. The flowchart which shows the procedure which determines an interpolation frame space | interval from decoding time information. The flowchart which shows the procedure which determines an interpolation frame space | interval from a user-specified interpolation frame space | interval. The flowchart which shows the procedure which controls the transmission frame space | interval of the moving image of a game terminal. The flowchart which shows the procedure in which a game terminal controls the transmission frame space | interval of a moving image by notifying provision of an image interpolation means to a game terminal. The flowchart of the process which changes the reception frame rate processing capability of a game terminal. The flowchart which shows the procedure in which a game terminal controls the transmission frame space | interval of a moving image by notifying the game terminal of the provision of an image interpolation means and the state of an image interpolation means. The flowchart of the process which changes the reception frame rate processing capability of a game terminal. The flowchart which shows the procedure which controls the transmission frame space | interval of the moving image of a game terminal, when recording a videophone. The flowchart which determines an interpolation frame space | interval from a user-specified interpolation frame space | interval when reproducing the recorded videophone. 3G-H. The block diagram which shows the structure of the conventional portable terminal in the videophone system using 324M.

Explanation of symbols

1000 H. 223 control unit 1010 MPEG4 / H. H.263 decoding unit 1011 decoded data storage unit 1020 image interpolation unit 1021 decoded image and interpolation image storage unit 1030 display unit 1040 245 control unit 1050 parameter determination unit 1060 MPEG4 / H. H.263 encoding unit 1070 imaging unit 1080 recording data storage unit 1101 transmission MPEG4 / H. 263 encoded data 1102 recorded data 1111 received MPEG4 / H.264 263 moving image encoded data 1112 decoding related parameters 1120 decoded data 1121 image interpolation control information 1131 decoded image data and interpolated image data 1141 H.245 message 1142 245 Control information 1151 Decoding time information 1152 Interpolated frame interval information specified by the user 1153 Transmission frame interval information requested from the terminal 1154 Image interpolation function ON / OFF information 1161 Encoding related parameters 1171 Imaging data 1181 Recording data

Claims (7)

  A portable terminal device constituting a videophone system configured by wireless connection, means for determining an optimum interpolation frame interval according to the processing capability of the own station, and interpolation according to the interpolation frame interval from decoded image data A portable terminal device comprising: means for generating image data.   2. The portable terminal device according to claim 1, wherein the means for determining the interpolation frame interval determines an optimal interpolation frame interval according to the processing capability of the local station based on the reception frame rate of the decoded image data.   The portable terminal device according to claim 1, wherein the means for determining the interpolation frame interval determines an optimal interpolation frame interval according to the processing capability of the own station based on interpolation frame interval information designated by a user.   The portable terminal device according to claim 1, further comprising means for notifying a receiving station of a reception frame rate capability set in the own station.   The portable terminal device according to claim 1, further comprising means for notifying a game of the presence / absence of means for generating the interpolated image data.   6. The mobile terminal device according to claim 5, further comprising means for notifying a game of an activity state of the means for generating the interpolated image data.   The portable terminal device according to claim 1, further comprising means for notifying a player of a transmission frame rate request.

Images