JP4871357B2 - Mobile device and video communication system - Google Patents

Description

  The present invention relates to a mobile video device, for example, a mobile terminal such as a PDA (Personal Digital Assistants) or a mobile phone with a camera, and more particularly to a mobile terminal having at least a video communication function and a video display function.

  In recent portable video devices such as PDAs, a function of rotating a video or an image on a display unit of the PDA to 0 degrees, 90 degrees, 180 degrees or the like as an OS is mounted. This is because there are various ways of holding a PDA that can be carried freely, such as portrait orientation, landscape orientation, and upside down. By implementing such a rotation display function, even if the PDA is held upside down, the display on the display unit can also be rotated upside down, and an image or image upright on the display unit can be obtained. Can be shown to the user.

  However, like a mobile phone terminal, a PDA is integrated with a video camera to display video from the video camera on its display unit, or videophone communication is performed on video or images from the video camera. When transmitting to the partner terminal, holding the PDA upside down has the disadvantage that the video or image is displayed upside down on the other party PDA or upside down on the other party performing videophone communication. The present invention provides a proposal for measures for dealing with such inconveniences.

  Note that the following [Patent Document 1] to [Patent Document 3] are known techniques related to the present invention. In the invention disclosed in [Patent Document 1], image rotation and conversion are performed on the image sending side.

  In the invention disclosed in [Patent Document 2], the direction data of the digital camera is incorporated into the image data at the time of photographing with the digital camera. When the image is reproduced and displayed later on a personal computer or the like, the direction data is read at the time of display, and the image to be displayed is rotated according to the direction data and displayed as an upright image.

  Furthermore, in the invention disclosed in [Patent Document 3], even when the imaging unit is rotated by a stomach camera or the like, the imaging unit is always directed in a predetermined direction by the action of gravity.

JP 2000-152069 A JP 2001-45354 A JP 62-65568 A

The three patent documents 1 to 3 described above have the following problems. First, with respect to Patent Document 1, when the broadcast transmission is performed to a plurality of receiving terminals or the video conferencing method is used such that images and images are communicated among a plurality of terminals, the invention There is a problem that
In general, it is desirable to be able to specify the display form of video and images according to the situation of the terminal that receives them. However, it is not possible to process image data on the transmission side as in the present invention. There is also a problem that it is inconvenient for the receiving side.

  Next, Patent Document 2 is an invention that is implemented in a closed environment where users can enjoy still images taken by themselves, such as transmitting images from a transmitting terminal to a receiving terminal via a network. There is a problem that it cannot be applied in some cases.

  Further, Patent Document 3 has a problem that it cannot be applied, for example, in a case where imaging data is transmitted from a transmitting terminal to a receiving terminal via a network, as in Patent Document 2.

Accordingly, the present invention has been made in view of the above problems, a video and / or image to disk press with mobile ObiSo location of the sender of the camera Tsuke携ObiSo placed al least one recipient via a network, to transmit upon, be rotated sender of the camera can be erect displayed to the receiving display, also further added also rotates and the receiving side of the display portable capable of erecting displayed on the display it is an object to provide a equipment. Furthermore, it is an object to provide a video communication system for a plurality chromatic their mobile ObiSo location.

  According to the present invention, as will be described later with reference to FIG. 4, the portable video device on the transmission side is a portable video device that sends video information on the transmission side, and (i) as the video information on the transmission side. A camera unit that captures an image to be transmitted to the receiving side, (ii) a rotation angle sensor unit that detects a rotation angle of the image when the above-described shooting is performed, and (iii) the detected rotation angle as rotation information. And a transmission unit that transmits the above-mentioned transmission side video information to the reception side.

  On the other hand, with respect to the mobile video device on the receiving side, receiving information including at least transmitting-side video information transmitted from the transmitting-side device and transmitting-side rotation information indicating the rotation angle of the video related to the transmitting-side video information is input. (I) a receiving unit that inputs the reception information, and (ii) transmission related to the transmission-side video information according to display data generated based on the reception information. A display unit that displays a side image, and (iii) a display data generation function unit that performs processing for generating the display data by using the reception information as input, and (iv) in the reception information It includes at least a transmission-side video memory that temporarily stores transmission-side video information, and (v) a calculation memory that temporarily stores transmission-side rotation information in the reception information.

  The reception-side portable video device preferably further includes (vi) a reception-side camera unit that is provided in the portable video device and transmits reception-side video information obtained by photographing on the reception side, and (vii) the portable video device. A reception-side rotation angle sensor unit that is provided in the apparatus and detects a rotation angle of the video associated with the reception-side video information when the above-described shooting is performed, and outputs the rotation angle as reception-side rotation information. And (viii) a receiving-side video memory for temporarily storing the receiving-side video information, and (ix) storing image (graphic) information to be generated and displayed on the display unit (ii) in the portable video device. An image memory. The reception side rotation information is temporarily stored in the arithmetic memory (v).

  Thus, according to the present invention, the effects shown in the following a) to d) can be obtained as will be apparent from the following description.

  a) When making a videophone call with a PDA or a mobile phone, even when these portable video devices are tilted, that is, when the device is not set upright, it is possible to display an upright image on the receiving device on the other side. .

  b) Even when sending images or images to multiple devices in a broadcast manner or when sending images or images between multiple devices, each device is not able to correct the images or the like regardless of the rotation angle. It can be displayed upright.

  c) Since the image or the like is rotated by the processing on the receiving side device, the display mode of the image or the like according to the preference of the receiving side user can be freely obtained.

  d) Even when displaying (monitoring) the video captured by the own device A, the same processing unit as that of the counterpart device B is provided on the display unit, so that the normal device and the own device can rotate correctly. A standing image or the like can be displayed. In addition, it is possible to add a process for mirror display of video etc. on the display unit of the receiving side device, but in this case, it can be processed regardless of the video on the transmitting side. The mirror processing can be done freely and easily.

  First, in order to speed up the understanding of the present invention, an example of a display pattern of an image or an image (hereinafter also simply referred to as an image) on the display unit of the video apparatus according to the present invention will be shown.

  FIG. 1 is a diagram showing a first display pattern of a transmission side video and a reception side video. In this figure, reference numeral 1 denotes a portable video device (hereinafter also simply referred to as a device), 2 denotes a camera unit, 3 denotes a display unit, and 4 denotes an operation unit such as a numeric keypad. Here, A attached to each reference number represents a video transmission side, and B represents a video reception side. Therefore, VA represents the transmission side image captured by the camera unit 2A, and VB represents the image displayed on the display unit 3B, that is, the reception side image.

  The first display pattern shown in FIG. 1 is a display pattern in the case where the present invention is not applied, and the transmission side image when the display object is photographed by the camera unit 2A with the apparatus 1 tilted (90 degrees in the example in the figure) is shown. Shown as VA. When the present invention is not applied, the video VA is displayed as a reception-side video VB tilted by 90 degrees on the screen of the display unit 3B of the upright reception-side device 1B as illustrated. Such a display pattern (VB) is not pleasant for the user on the receiving side.

  FIG. 2 is a diagram showing a second display pattern to which the present invention is applied. Throughout the drawings, similar components are denoted by the same reference numerals or symbols.

  In FIG. 2, 5A is a rotation angle sensor unit, which outputs the rotation information Ir (rotation) A (= 90 °) of the apparatus 1A tilted 90 degrees and transmits it to the apparatus 1B together with the video information Iv (video) A. Is done. At this time, the video VA received at an inclination becomes an upright video VB based on the rotation information IrA.

  FIG. 3 is a diagram showing a third display pattern to which the present invention is applied. In FIG. 2, this figure shows a case where the receiving side device 1B itself that has received the rotation information IrA and the transmitting side video information IvA from the device 1A is tilted. For example, if it is now tilted 90 degrees (IrB = 90 °), the rotation angle obtained by combining this IrB and the received IrA generates the upright video VB as shown in FIG. "Instruction" and "rotation processing" are executed in the apparatus 1B.

  As described above, the explanation for speeding up the understanding of the present invention has been given, and a more specific explanation of the present invention will be given below. First, referring to FIG. 4, this figure is a diagram showing a basic configuration of a portable video apparatus (transmission side) according to the present invention.

  In FIG. 4, reference numerals 2A and 5A are a camera unit and a rotation angle sensor unit, respectively, and 6A is a transmission for transmitting these pieces of output information to the reception-side apparatus 1B via the wireless / wired network NW. Part. That is, the apparatus (transmission side) 1A according to the present invention is a portable video apparatus that transmits transmission-side video information, and a camera unit 2A that captures a video VA to be transmitted to the reception side (1B) as transmission-side video information IvA. And a rotation angle sensor unit 5A that detects the rotation angle of the video when the image is taken, and a transmission unit that transmits the detected rotation angle as rotation information IrA to the reception side (1B) together with the transmission side video information IvA. 6A is a portable video device.

  Here, the rotation angle sensor unit 5A that generates the rotation information IrA can be realized by any of a gyro sensor, a gravity sensor, and a tilt sensor. 2 and 3 described above show a gyro sensor as an example. Some of these gyro sensors, gravity sensors, and tilt sensors are already commercially available and inexpensively available. Further, Patent Documents 1 to 3 described above also disclose rotation angle sensor units that are devised independently.

  Further, the transmitter 6A in FIG. 4 will be described in detail. The transmission unit 6A includes (i) a one-to-one communication function for transmitting transmission-side video information IvA and rotation information IrA to one reception-side device 1B, and (ii) transmission-side video information IvA and a plurality of reception-side devices 1B. A communication function of at least one of the one-to-many communication functions for sending the rotation information IrA can be provided. FIG. 5 is a diagram illustrating an example of distribution by the transmission unit 6A having the above-described point-to-multipoint communication function. In this figure, only one direction from the device 1A to each device 1B is shown, but there is also reverse transmission from each device 1B to the device 1A.

  In FIG. 5, the apparatus 1A transmits information (IvA) such as video, image, and audio from the transmission unit 6A using an RTP packet, and transmits rotation information (IrA) using an RTCP packet. These transmissions are preferably performed toward each of the devices 1B1 to 1B3 on an IP (Internet Protocol) network NW. As already known, the above protocol RTP / RTCP (Real-time Transport Control Protocol) is defined in IETF (http://www.ietf.org/)RFC1889, and information such as video by RTP. The transmission is controlled by RTCP.

  FIG. 6 is a diagram showing the structure of the APP packet field in RTCP. There are several types of RTCP packets, and the rotation information IrA according to the present invention uses APP (packets based on application-specific rules) in the types. Generally, a protocol for sending video and data has a mechanism for sending the additional information. Therefore, the rotation information IrA can be sent as the additional information. Further, according to FIG. 6, frequency information IfA (frequency) is further added as the additional information.

As described above, the transmission unit 6A according to the present invention is characterized in that the transmission side video information IvA is transmitted according to RTP for data transmission, and the rotation information IrA is transmitted according to RTCP for control.
In addition to the rotation information IrA, the control packet according to the RTCP further includes frequency information IfA indicating the frequency of updating the rotation information.

  Details of the field configuration shown in FIG. 6 are as follows.

V (1 bit length): Product version based on RFC1889 (= 2).
P (1 bit length): indicates whether there is a padding octet attached to the end of the packet.
Subtype (5-bit length): Fixed to 0.
PT (8-bit length): Indicates the type of RTCP packet as a packet type. 204 fixed.
Length (16-bit length): The total length of the RTCP packet divided by 32 and subtracted by 1.
Sender SSRC (32-bit length): Sender identification number name: Name indicating rotation information RTCP packet: “kait”
Rotation information (16-bit length): rotation angle of the device. Up to 360 degrees clockwise with 0 degrees upright.
Rotation information update frequency (16-bit length): Time interval for updating rotation information. 0.01 second unit.

  The receiving unit 7B in each of the receiving side devices 1B1 to 1B3 (FIG. 5) recognizes the type of received RTCP packet, rotation information, and the like by referring to the PT field and the name field. . Note that other communication protocols may be used to implement the present invention.

  Next, with reference to FIG. 7, a basic configuration of the portable video apparatus (reception side) according to the present invention is shown. In FIG. 7, reference numeral 7B is the receiving unit described above, 8B is a memory function unit, 9B is a display data generation function unit, and 3B is the display unit described above.

  That is, the apparatus (reception side) 1B according to the present invention at least receives transmission-side video information IvA sent from the transmission-side apparatus 1A and transmission-side rotation information IrA indicating the rotation angle of the video related to the transmission-side video information. A portable video device that receives received information Iin, the receiving unit 7B that receives the received information Iin, a memory function unit 8B that temporarily stores the received information Iin received by the receiving unit 7B, and the memory In accordance with display data Dd (display) generated based on the reception information Iin stored in the function unit 8B, the display unit 3B that displays the transmission side video VA related to the transmission side video information IvA, and the reception information Iin described above And a display data generation function unit 9B that performs processing for generating display data Dd.

  Here, the memory function unit 8B (FIG. 7) includes a transmission-side video memory 81 that temporarily stores the transmission-side video information IvA in the reception information Iin, and an arithmetic memory that temporarily stores the transmission-side rotation information IrA in the reception information Iin. 82.

  Further, the display data generation function unit 9B (FIG. 7) sends the transmission side rotation information IrA stored in the arithmetic memory 82 to the original scan address AD (see FIGS. 9 and 10) for performing the sequential scan in the display unit 3B. The transmission side video address conversion unit 91 for generating the conversion scan address adA (see FIGS. 9 and 10) to which the address conversion corresponding to the above is added.

  In the portable video device 1 according to the present invention, the receiving-side device 1B itself also transmits the receiving-side video VB captured by its own camera unit 2B and / or the image GB related to the image information Ig (graphic) created by itself. It can be displayed on the display unit 3B together with the side video VA. FIG. 8 is a diagram showing a basic configuration of another portable video apparatus (reception side) according to the present invention, and the video VB and the image GB can be displayed on the display unit 3B together with the video VA.

  That is, the portable video terminal 1B is provided with the receiving camera unit 2B that sends out the receiving-side video information VB obtained by shooting at the receiving side, and the reception that is provided in the portable video device 1B when shooting is performed. A reception-side rotation angle sensor unit 5B that detects the rotation angle of the image related to the side image information IvB and outputs the detected rotation angle as reception-side rotation information IrB. Then, the memory function unit 8B further receives a reception-side video memory 83 that temporarily stores the reception-side video information IvB, and image (graphic) information Ig to be generated by the CPU 11 in the portable video apparatus 1B and displayed on the display unit 3B. And the receiving side rotation information (IrB) is temporarily stored in the arithmetic memory 82 described above.

  The display data generation function unit 9B receives the original scan address AD as it is for the image memory 84 and converts the address corresponding to the reception side rotation information IrB stored in the arithmetic memory 82. A receiving side video address conversion unit 92 that generates a conversion scan address adB (see FIG. 9 and FIG. 10) to which is added is included.

  Furthermore, the display data generation function unit 9B combines the display data from the transmission-side video memory 81, the reception-side video memory 83, and the image memory 84 in the memory function unit 8B, and sends it to the display unit 3B. A data synthesis unit 93 is included.

A more specific configuration of the basic configuration (FIGS. 7 and 8) of the apparatus 1B will be described above. FIG. 9 is a diagram showing a state of address conversion in the receiving side device.
FIG. 10 is a diagram illustrating a specific configuration example of the entire receiving-side apparatus.
FIG. 11 is a diagram showing a video and an image generated by the receiving side itself in the receiving side device.

  First, referring to FIG. 9, the video information IvA transmitted from the camera unit 2A is stored in the video memory 81 while being overwritten frame by frame. Also, video information IvB for each frame from its own camera unit 2B, as well as image information Ig that is made up of characters and graphics (video description, video frame, etc.) generated by its own CPU (11 in FIG. 10). They are stored in the video memory 83 and the image memory 84, respectively. The audio information is not related to the present invention and is omitted.

  The information (IvA, IvB, Ig) stored in these memories (81, 83, 84) is sequentially read according to the original scan address AD from the read address generation unit 12. The read information (IvA, IvB, Ig) is synthesized as display data DdA, DdB, and DdG by the display data synthesis unit 93, and then becomes final display data Dd. Provided to the user as a display screen.

  The above operation is a case where there is no rotation in either the device 1A or the device 1B. If these devices 1A and 1B are rotated, the rotation information IrA and IrB based on the present invention acts on the address conversion units 91 and 92, respectively, and the original scan address AD is converted into the converted scan, respectively. After conversion into the addresses adA and aDB, the corresponding memories 81 and 83 are read out. Here, an upright screen display is made regardless of the above rotation. Note that the display data DdG based on the image information Ig is irrelevant to the camera unit 2B, and thus the address conversion due to the rotation is not necessary.

  Next, referring to FIG. 10, the CPU 11, the receiving unit 7 </ b> B having the antenna 71 and the transmission interface 72, the arithmetic memory 82, and the rotation angle sensor unit 5 </ b> B that are not described in FIG. 9 are illustrated in FIG. 10. ing.

  In the configuration of FIG. 10, when only the portion related to the video IvB and the image Ig (GB) generated by the receiving side device 1B itself is extracted, the result is as shown in FIG.

  The above FIG. 10 will be described in a little more detail. FIG. 12 is a flowchart showing an example of a method of dividing the video information IvA and the rotation information IrA in FIG.

Step S11: The transmission interface 72 of the receiving unit 7B shown in FIG. 10 receives the packet forming the reception information Iin (the RTP / RTCP packet described with reference to FIG. 6).
Step S12: Look at the identifier (RTP / RTCP) of the packet to determine whether it is video information (RTP) or rotation information (RTCP),
Step S13: If it is RTP, the compressed video information IvA is decoded and stored in the video memory 81.

Step S14: If it is RTCP, that is, if it is rotation information IrA, the rotation angle to be adjusted is calculated in consideration of its own rotation information IrB. The calculation result is reflected in the address conversion unit 91.
Step S15: Further, based on the own rotation information IrB, the memory 83 is read with the address converted by the address conversion unit 92 so as to erect the own video VB.

  Next, “address conversion” will be described in more detail. First, let us consider the original scan address (image display address) for the transmission-side video memory 81 shown in FIG. That is, it is address conversion for displaying the contents of the video memory 81 at a desired position on the display unit 3B.

Here, when the original scan address AD is expressed as (x, y) and the address to be given to the video memory is expressed as (P, Q), it can be expressed by the following equation (1).
P = ax + by + c
Q = dx + ey + f (1)
However, a, b, c, d, e, and f are coefficients (constants) for setting in which coordinate on the display unit 3B and in which size the video VA is to be expressed, and the CPU 11 uses predetermined application software. Specify by. The address conversion based on the equation (1) is performed in the transmission side video address conversion unit 91. Note that when the value of (P, Q) exceeds the value of the address of the video memory 81, the address conversion is not performed. Further, when there is a portion that is not desired to be displayed, transparent color data Dt (transparent) is written in the memory 81.

FIG. 13 is a diagram schematically showing address conversion for displaying the display data DdA of (B) at a predetermined position (A) on the display unit 3B. Assuming that the video in the rectangular area “R” on the video data DaA shown in FIG. 5B is displayed in the lower left area “Z” on the screen (A) of the display unit 3B, the above equation (1) is It is represented by the following formula (2). In this example, the CPU 11 sets the above-described coefficients a to f to a = 20, b = 0, c = 100, d = 0, e = 20 and f = 200 by calculation.
P = 20x + 100
Q = 20y + 200 (2)

  Next, address conversion considering rotation information (IrA, IrB) which is a point of the present invention will be described. FIG. 14 is a diagram schematically showing address conversion for displaying (A) the display data DaA accompanying the rotation of (B) upright on the display unit 3B. That is, in (B) of this figure, “R” is the image of the other party (transmission side) and is rotated by the rotation angle θa. This is corrected to an upright video “Z” as shown in FIG.

  How the video VA from the other party (transmission side) is rotated can also depend on the preference of the receiving user to display. For example, when viewing a sleeping image, the other party's image may not be upright with respect to gravity, and it may be more convenient if the image is in the same direction as the device itself. Therefore, the rotation angle may be changed under the user's adjustment instead of always automatically erecting. At this time, the CPU 11 calculates a value based on a setting value input by the user operating the operation unit 4B as the above-described coefficients a to f.

  In FIG. 14, the coordinates of the center point of the rectangle “Z” are ((x0 + x1) / 2, (y0 + y1) / 2), and the center point is rotated by the rotation angle θa to be erect. That is, the rotation information (IrA) sent from the other party (transmission side) is IrA = θa.

Here, the display coordinates (x, y) on the display unit 3B are expressed by the following formula (3).
P (A) = {x− (x1−x0 + 1) / 2} cos θa− {y− (y1−y0 + 1) / 2} sin θa + (x0 + x1) / 2
Q (A) = {x− (x1−x0 + 1) / 2} sin θa + {y− (y1−y0 + 1) / 2} cos θa + (y0 + y1) / 2 (3)

  In this equation (3), the rotation angle θa is used for easy understanding. However, in the end, not only enlargement, reduction, movement, and trimming but also rotation can be expressed by the above-described equation (1).

If it becomes easy to see the said Formula (3), it will become like the following Formula (4).
P (A) = {x− (x1−x0 + 1) / 2} cos θa− {y− (y1−y0 + 1) / 2} sin θa + (x0 + x1) / 2
Q (A) = {x− (x1−x0 + 1) / 2} sin θa− {y− (y1−y0 + 1) / 2} cos θa + (y0 + y1) / 2 (4)
When the above equation (4) is further modified, the following equation (5) is obtained.
P (A) = cos θa · x-sin θa · y-Cpa
Q (A) = sin θa · x-cos θa · y-Cqa (5)
Here, the coefficients a, b, c, d, e, and f in the above equation (1) are as follows.
a = cos θa
b = −sin θa
c = {(x1−x0 + 1) cos θa + (y1−y0 + 1) sin θa + x0 + x1} / 2 = Cpa
d = sin θa
e = −cos θa
f = {(x1-x0 + 1) sin θa + (y1-y0 + 1) cos θa + y0 + y1} / 2 = Cqa

  Thus, the transmission side video address conversion unit 91 (FIG. 10) assumes that the original scan address AD is (x, y) and the transmission side rotation angle indicated by the transmission side rotation information IrA is θa. (A) and Q (A)) are calculated by the above equation (5). Here, Cpa and Cqa are constants including cos θa and sin θa, respectively.

  The address (x, y) is given so as to scan from left to right, left to right,... From the upper left on the screen of the display unit 3B, for example. The addresses P (A) and Q (A) are calculated by the above equation (5), and the PQ plane is scanned obliquely (rotation angle θa).

  When the address (P (A), Q (A)) calculated by the above equation (5) based on the address (x, y) exceeds the access range of the video memory 81, the memory 81 is read. Address is not given, and the read data from the memory 81 is transparent color data Dt (transparent).

Further, the address (P (A), Q (A)) calculated by the above equation (5) based on the address (x, y) may be cut out beyond the range of the video memory 81. . That is, the coordinates of the end of the xy plane, that is, the end of the display screen (3B) are in the middle of the PQ plane. In such a case, a part of the video on the PQ plane is not output and is not displayed (the video is cut off).
Therefore, when the image is rotated and the size does not fit on the display screen, the image may be cut off at the edge of the screen or the partner image may be interrupted in the middle of the screen.

  The case using the above formulas (3) to (5) is a case where only the transmission side rotation angle θa is considered, but when the reception side rotation angle θb is also considered (the following (I) ) On the other hand, with respect to the video on the receiving side, there is a case where only the receiving side rotation angle (this is distinguished from θb and set to θc) is considered ((II) below). Furthermore, there is a case of a mirror display (FIGS. 18 and 19) described later.

(I) The transmission-side video address conversion unit 91 sets the conversion scan address (P (A), Q (A)) to the following equation (6), where θb is the reception-side rotation angle indicated by the reception-side rotation information IrB. ).
P (A) = cos ([theta] a- [theta] b) * x-sin ([theta] a- [theta] b) * y-Cpa
Q (A) = sin ([theta] a- [theta] b) * x-cos ([theta] a- [theta] b) * y-Cqa (6)
Here, the coefficients a, b, c, d, e, and f in the above equation (1) are as follows.
a = cos (θa−θb)
b = −sin (θa−θb)
c = {(x1−x0 + 1) cos (θa−θb) + (y1−y0 + 1) sin (θa−θb) + x0 + x1} / 2 = Cpa
d = sin (θa−θb)
e = −cos (θa−θb)
f = {(x1−x0 + 1) sin (θa−θb) + (y1−y0 + 1) cos (θa−θb) + y0 + y1} / 2 = Cqa

(II) The reception side address conversion unit 92 assumes that the original scan address is (x, y) and the reception side rotation angle indicated by the reception side rotation information IrB is θc, and the conversion scan address (P (C), Q (C)) is calculated by the following equation (7).
P (C) = cos2θc · x-sin2θc · y-Cpc
Q (C) = sin2θc · x-cos2θc · y-Cqc (7)
Here, the coefficients a, b, c, d, e, and f in the above equation (1) are as follows.
a = cos 2θc
b = −sin2θc
c =-{(x1-x0 + 1) cos2 [theta] c + (y1-y0 + 1) sin2 [theta] c + x0 + x1} / 2 = Cpc
d = sin2θc
e = −cos 2θc
f = − {(x1−x0 + 1) sin2θc + (y1−y0 + 1) cos2θc + y0 + y1} / 2 = Cqc
The reason for setting 2θc will be described later (FIG. 17).

The state of the address conversion described above is illustrated schematically as shown in FIGS. First,
(A) and (B) of FIG. 15 are diagrams showing a case where the counterpart device is inclined by the rotation angle θa.
(A) and (B) of FIG. 16 are diagrams showing a case where the device itself is also inclined by the rotation angle θb.
FIGS. 17A, 17B, and 17C are diagrams showing a case where the image of the device is tilted by the rotation angle θc. FIGS. 15, 16, and 17 are respectively described above. (5), (6), and (7).

  In FIG. 15A, 21 schematically represents the contents of the memory 81, and is the contents of the image VA captured when the counterpart apparatus 1A is tilted by θa. In addition, 22 is a mere mark for making the direction of inclination easy to understand. The video data when the image of FIG. 15A is received by the own device 1B, that is, the screen display when the own device 1B is upright is as shown in FIG.

  Next, in FIG. 16, when the own device is inclined by θb, the screen display is as shown in FIG. 16A, and is inclined by (θa−θb). When this video data is rotated by (θa−θb), the screen display is upright, as shown in FIG.

  Further, referring to FIG. 17, a state in which the display target is photographed by the camera unit 2 </ b> B of the apparatus 1 </ b> B is shown in FIG. Since the camera is facing away, the back of the apparatus is visible (the display unit 3B is also away). If the rotation angle of the device 1B at this time is θc, the image is tilted by −θc with respect to the camera as shown in FIG.

  However, since the device 1B itself is actually inclined by θc, the image is inclined by θc × 2 (= θc + θc) as shown in FIG. “2θc” in the above-described equation (7) means this.

  Next, mirror display will be described. When viewing a video of yourself taken with the camera 2B, it is natural that there is no sense of incongruity because the person who flips the left and right of the video looks like a mirror.

  18A to 18E are views for explaining mirror display based on the present invention. This figure (A) is the same as FIG. 17 (A) described above, and it is assumed that the apparatus 1B is rotated counterclockwise by the rotation angle θc when viewed from the back of the apparatus 1B. Then, when viewed from the camera unit 2B of the apparatus 1B, the image appears to be tilted by −θc clockwise as shown in FIG. When this image is inverted with respect to the vertical axis passing through the camera (2B), the result is as shown in FIG. When this is displayed on the screen on the display unit 3B, it is as shown in FIG. However, in reality, the apparatus 1B is also tilted by −θc, and the tilt of the video is canceled as shown in FIG. 18E, so that the video appears to be upright without address conversion.

FIG. 19 is a diagram showing an area (B) to be displayed in “mirror display” and an image (A) displayed on the display unit 3B, and corresponds to the above-described FIG. 13 and FIG. That is, when the user's own image taken by the camera unit 2B in FIG. 10 is displayed on his / her device 1B, the image is displayed as a mirror image obtained by inverting the left and right of the image. An image captured from this camera is on the PQ plane shown in FIG. 19B, and is converted into an image on the xy plane shown in FIG. 19A. In this display, it is assumed that the PQ plane is in the reception-side video memory 83 in FIG. 10, and the screen for displaying it is the xy plane. When scanning the xy plane, when the coordinates to be displayed are (x, y), an expression indicating what value the coordinate (P, Q) of the PQ plane takes is a coordinate conversion formula. As a general formula, the above-described formula (1) is obtained, and in the example of FIG. 19, the following formula (8) is obtained.
P = 10x + 300
Q = 20y + 200 (8)

  To summarize the above mirror display, when the display side 3B is instructed to display the mirror on the receiving side video information IvB by the receiving side camera unit 2B, the above-mentioned receiving side address conversion unit 92 (FIG. 10). In this case, only the left and right interchange of the original scan address (x, y) is executed without performing the conversion to the conversion scan address.

  Finally, the display data composition unit 93 shown in FIGS. 9 and 10 will be described in more detail. FIG. 20 is a diagram simply illustrating the operation concept of the display data synthesis unit 93, and FIG. 21 is a diagram illustrating the concept of FIG. 20 with a specific example.

  First, referring to FIG. 20, the display data synthesis unit 93 includes a video VA from the other party (transmission side terminal 1A), a video VB generated within itself (reception side terminal 1B), and an image GB generated within itself. Are combined and displayed as a composite image / image V on the screen of the display unit 3B. A specific example is shown in FIG.

  In FIG. 21, the upper left diagram shows the partner video (VA) whose periphery is filled with the transparent color data Dt, and the middle left diagram shows the self (1B) video VB with the transparent color data Dt, for example. This figure shows a graphic image (GB) generated within itself, and when these are combined, a composite image / image V shown in the right figure is obtained.

  As described above, as a result of performing the address conversion, there may be a case where an image that protrudes from the frame, a missing image, or the like appears. In such a case, it is preferable that the protruding portion and the missing portion are not displayed black but are filled with the transparent color data Dt. Referring to FIG. 20 again, if the display data # 101 is the transparent color data Dt, the display data # 102 below is used. If the display data # 102 is the transparent color data Dt, the display data # 103 below the display data # 102 is used, and when these composites (# 101 + # 102 + # 103) are viewed from above, composite display data # 104 is obtained.

  Summarizing the above, the display data synthesizing unit 93 (FIG. 10) causes the transmission side video memory 81 and the reception side video respectively corresponding to the respective address translations by the transmission side address translation unit 91 and the reception side address translation unit 92. When reading of each display data from the memory 83 becomes impossible and the display data from these memories becomes no data, the function of replacing the no data with transparent color data and sending it to the display unit 3B is achieved.

  More preferably, when the display data composition unit 93 displays the display data from the transmission side video memory 81, the reception side video memory 83, and the image memory 84 so as to overlap each other in the depth direction from the front of the display unit 3B, the display data composition unit 93 arranges the display data. A priority order setting unit for setting the order is provided. The configuration for this is shown in the following figure.

  FIG. 22 is a diagram showing an example of the configuration of the display data synthesizing unit 93, and the above priority order setting unit is indicated by reference numeral 104 in the upper center of the figure. The priority order setting unit 104 designates contact connection positions (that is, priority order) for the three selection switches 101, 102, and 103. The command is given from the CPU 11 (FIG. 10).

  The contact connection positions are indicated by “1”, “2”, and “3” in each of the selection switches 101, 102, and 103, and “1” → “2” → “3”. The priority becomes lower in the order of. According to FIG. 22, VA from the transmission-side video memory 81 is given to each “1” contact, and VB from the reception-side memory 83 is given to each “2” contact. GB from the image memory 84 is given to the contact. That is, first, the video VA is displayed with the highest priority. If a part of this VA is replaced with the transparent color data Dt, the next priority VB is displayed in that part. If a part of this VB is replaced with the transparent color data Dt, the GB of the next priority is displayed in that part.

  That is, according to the configuration of FIG. 22, when the transparent color data Dt (for example, “FF” in hexadecimal notation) is detected within the VA by the transparent color data detection unit 105, the contact “4” of the selection switch 106 is obtained. Is switched from “5” to “5”, and the display data given through the contact “5” is sent to the display unit 3B.

  The display data given through the contact “5” is the display data obtained through the contact “6” in the selection switch 108, that is, the reception side video VB. However, if the transparent color data Dt is included in a part of the VB, the transparent color data detection unit 107 that detects this Dt switches the contact in the selection switch 108 from “6” to “7”. The GB given through the contact “3” in 103 is sent to the display unit 3B via the contact “7” → the contact “5”. Thus, for example, the composite display / image V shown in FIG. 21 is sent to the display unit 3B and displayed on the screen.

The present invention described above also provides (1) a video display system, (2) a video display method, and (3) a video display program.
(1) The video display system
A portable video device for transmitting video information on a transmitting side, a camera unit for capturing a video to be transmitted to a receiving side as video information on the transmitting side, and a rotation for detecting a rotation angle of the video when performing the above photographing A transmission-side portable video device comprising: an angle sensor unit; and a transmission unit that transmits the detected rotation angle to the reception side together with the transmission-side video information as rotation information. A reception unit that inputs reception information including transmission-side video information and transmission-side rotation information indicating the rotation angle, and transmission according to the transmission-side video information according to display data generated based on the reception information A reception-side portable video device comprising: a display unit that displays a side image; and a display data generation function unit that performs processing for generating the display data using the reception information as an input. The

(2) The video display method is
In the transmission-side portable video device that transmits the transmission-side video information,
Shooting a video to be sent to the receiving side as the sending side video information with a camera unit;
Detecting a rotation angle of the image at the rotation angle sensor unit when performing the photographing;
Transmitting the detected rotation angle as rotation information together with the transmission-side video information from the transmission unit to the reception side,
In the reception-side portable video device that receives the reception information including the transmission-side video information sent from the transmission-side device and the transmission-side rotation information indicating the rotation angle,
Inputting the reception information at a receiving unit;
In accordance with display data generated based on the reception information, displaying a transmission side video according to the transmission side video information on a display unit;
And a process of generating the display data by a display data generation function unit using the received information as an input.

(3) -1 The program on the sending side is
In the transmission-side portable video device that transmits the transmission-side video information,
A procedure for photographing a video to be transmitted to the reception side as the transmission side video information;
A procedure for detecting a rotation angle of the video when performing the shooting;
And a procedure for transmitting the detected rotation angle as rotation information together with the transmission-side video information to the reception side.

(3) -2 The program on the receiving side is
In a reception-side portable video device that receives reception information including at least transmission-side video information transmitted from a transmission-side device and transmission-side rotation information indicating a rotation angle of a video related to the transmission-side video information.
Inputting the received information;
In accordance with display data generated based on the reception information, a procedure for displaying a transmission side video according to the transmission side video information;
And a procedure for performing a process for generating the display data using the reception information as an input.

As detailed above, according to the present invention,
When the transmission device transmits image data taken by the camera with the device tilted, an automatically erect image can be provided on the display unit of the reception user device.
The above-mentioned video upright can be realized by point-to-point communication or point-to-multipoint communication.
It is possible to provide an upright image to the user device in consideration of the tilt angle in the receiving device.

It is a figure which shows the 1st pattern of a transmission side image | video and a receiving side image | video. It is a figure which shows the 2nd display pattern to which this invention is applied. It is a figure which shows the 3rd display pattern to which this invention is applied. It is a figure which shows the basic composition of the portable video apparatus (transmission side) which concerns on this invention. It is a figure which shows the example of a delivery by the transmission part provided with the one-to-many communication function. It is a figure which shows the structure of the APP packet field in RTCP. It is a figure which shows the basic composition of the portable video apparatus (reception side) which concerns on this invention. It is a figure which shows the basic composition of the portable video apparatus (reception side) which concerns on this invention. It is a figure which shows the mode of address conversion in a receiving side apparatus. It is a figure which shows the specific structure of the whole receiving side apparatus. It is a figure shown about the image | video and image which the receiving side itself produces | generates in the receiving side apparatus. It is a flowchart which shows an example of the method of the division | segmentation of the video information in FIG. 10, and rotation information. It is a figure showing the address conversion which displays the display data DdA of (B) on the predetermined position (A) on the display part 3B. It is a figure showing address conversion which displays (A) the display data DdA accompanied by rotation of (B) upright on a display part. (A) And (B) is a figure which shows the case where the other party inclines only rotation angle (theta) a. (A) And (B) is a figure which shows the case where the own apparatus also inclines only rotation angle (theta) b. (A), (B), and (C) are diagrams showing a case where the image on the device is inclined by the rotation angle θc. (A)-(E) are the figures for demonstrating the mirror display based on this invention. In "mirror display", it is a figure which shows the area | region (B) to display, and the image | video (A) displayed on a display part. It is a figure which simply represents the operation | movement concept of a display data synthetic | combination part. FIG. 21 is a diagram illustrating the concept of FIG. 20 with a specific example. It is a figure which shows the example of 1 structure of a display data synthetic | combination part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Portable video apparatus 2 Camera part 3 Display part 4 Operation part 6 Transmission part 7 Reception part

Claims (8)

In the mobile device that receives the reception information including at least the transmission-side video information transmitted from the transmission-side mobile device and the transmission-side rotation information indicating the rotation angle of the video related to the transmission-side video information,
A reception unit that receives the reception information from the transmission side; and
A camera unit that sends video information on the receiving side by shooting on the receiving side;
A rotation angle sensor unit that detects a rotation angle of an image related to the reception-side video information and outputs the rotation angle as reception-side rotation information;
A display unit that displays at least the video associated with the transmission-side video information and the reception-side video information via a memory function unit and a display data generation function unit by sequential scanning, and
The memory function unit
A transmission side video memory for storing the transmission side video information;
A receiver video memory for storing the receiver video information;
An arithmetic memory for storing the reception side rotation information,
The display data generation function unit
With respect to the transmission side video information, a transmission side video address conversion unit that performs address conversion corresponding to a rotation angle based on the reception side rotation information stored in the arithmetic memory with respect to the original address for the sequential scan,
With respect to the receiving side video information, a receiving side video address conversion unit that performs address conversion corresponding to the rotation angle with respect to the original address;
Display data read from the transmission-side video memory at the address converted by the transmission-side video address conversion unit and display data read from the reception-side video memory at the address converted by the reception-side video address conversion unit A portable data device comprising: a display data synthesizing unit that synthesizes both data and sends them to the display unit . An image memory for storing graphic information displayed on the display unit is further included, and the graphic information read from the image memory at the original address is synthesized with the display data in the display data synthesizing unit. The portable device according to claim 1, which is sent out . The display data synthesizing unit is configured to display each display data from the transmission side video memory and the reception side video memory corresponding to each of the display side video address conversion unit and the reception side video address conversion unit. when the reading becomes impossible display data from these image memories it becomes no data, to claim 1 or 2, characterized in that sending to the display unit replaces the radio data to the transparent color data The portable device described . The display data composition unit sets the arrangement order when displaying the display data from the transmission side video memory, the reception side video memory, and the image memory so as to overlap each other in the depth direction from the front of the display unit. The portable device according to claim 2 , further comprising a priority order setting unit. A transmission-side portable device that transmits transmission information including at least transmission-side video information and transmission-side rotation information indicating a rotation angle of a video related to the transmission-side video information;
A receiving-side portable device that receives the transmission information from the transmitting-side portable device;
A reception unit that receives the reception information from the transmission side, a camera unit that transmits reception-side video information obtained by shooting on the reception side, and a rotation angle of a video associated with the reception-side video information that is detected by the reception side A rotation angle sensor unit that outputs as rotation information, and a display unit that displays at least the video on the transmission side video information and the reception side video information by sequential scanning via the memory function unit and the display data generation function unit, Including and
The memory function unit includes a transmission-side video memory that stores the transmission-side video information, a reception-side video memory that stores the reception-side video information, and an arithmetic memory that stores the reception-side rotation information.
The display data generation function unit performs transmission to perform address conversion corresponding to a rotation angle based on the reception side rotation information stored in the arithmetic memory with respect to the original address for the sequential scan with respect to the transmission side video information. A side video address conversion unit, a reception side video address conversion unit that performs address conversion corresponding to the rotation angle with respect to the original address with respect to the reception side video information, and an address converted by the transmission side video address conversion unit The display data read out from the transmission side video memory and the display data read out from the reception side video memory at the address converted by the reception side video address conversion unit are combined and sent to the display unit. A receiving side portable device comprising:
A video communication system comprising: An image memory for storing graphic information displayed on the display unit is further included, and the graphic information read from the image memory at the original address is synthesized with the display data in the display data synthesizing unit. 6. The video communication system according to claim 5, which is transmitted. The display data synthesizing unit is configured to display each display data from the transmission side video memory and the reception side video memory corresponding to each of the display side video address conversion unit and the reception side video address conversion unit. 7. When the display data from the memory becomes no data because the data cannot be read out, the no data is replaced with transparent color data and sent to the display unit. Video communication system. The display data composition unit sets the arrangement order when displaying the display data from the transmission side video memory, the reception side video memory, and the image memory so as to overlap each other in the depth direction from the front of the display unit. The video communication system according to claim 6, further comprising a priority order setting unit.