JP2004128614A - Image display controller and image display control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display the image of main data and an image of subordinate data under an appropriate state at all times while preventing superposition or deviation from a display screen. <P>SOLUTION: An object image is cut out from image data received from client terminals T1 and T2 and its arranging position on the screen is determined. Furthermore, a balloon image indicative of the content of a text is generated from text data received from the client terminals T1 and T2 and then the arranging position of the balloon image is determined by a source in the vicinity of the object image. If the balloon image deviates from the screen or is superposed on another image, position of the balloon image is adjusted not to deviate from the screen or not to be superposed on another image before a synthesized image is formed. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image display control device and an image display control program used in a system in which a plurality of communication terminal users can communicate in real time while sharing images.
[0002]
[Prior art]
2. Description of the Related Art Communication systems (hereinafter referred to as chats) are used in which a plurality of communication terminals are connected to each other via a communication network using the Internet or the like so that users of the communication terminals can have a conversation in real time. In this chat, generally, text data input by each user is displayed on a display screen of each communication terminal to have a conversation.
[0003]
On the other hand, as a further development of the above-mentioned chat, in each communication terminal, for example, a still image or a moving image in which a face of a user is photographed is synthesized and displayed on a common background image using a camera or the like, and further, each user is displayed. A system has been proposed in which text data input by a user is displayed as a so-called balloon image on the composite image to have a conversation (for example, see Non-Patent Document 1).
[0004]
[Non-patent document 1]
Takashi Ida et al., "Realization of Chat System with Video Sharing Using Real-time Face Image Extraction and MPEG-4", Proceedings of the Intelligent Information Media Symposium, December 2000, p. 13-18
[0005]
[Problems to be solved by the invention]
However, in the communication system exemplified in the literature, the display position of the balloon image is set so as to always have a fixed positional relationship with the display position of the user's face image. For this reason, a part of the balloon image may protrude from the display screen, or may be displayed in a state of being superimposed on the face image of the user, and the text data input by the user may not be accurately transmitted to the user of the communication partner. .
[0006]
The present invention has been made in view of the above circumstances, and an object thereof is to ensure that an image of main data and an image of subordinate data are always displayed in an appropriate state without overlapping or protruding from a display screen. It is another object of the present invention to provide an image display control device and an image display control program which enable accurate and smooth chat communication.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the present invention relates to an image display control device provided in a server device or a terminal device interposed between a plurality of terminal devices and a display control program therefor, and an adjusting means for adjusting a form of a composite image Is newly provided. When the received main data and the dependent data are combined to generate combined data that can be displayed on one display screen, the received main data image and the dependent data image fall within the display screen area. The combined form is adjusted so as not to overlap each other, and the combined data adjusted in the combined form is transmitted to the terminal device of the communication partner and displayed.
[0008]
Therefore, according to the present invention, at the time of composition, the composition form of the image of the main data and the image of the dependent data, for example, the position, size, and shape are always displayed in an appropriate state. Therefore, no matter where the composite image is displayed on the display screen, and regardless of the size or shape of the image of the main data, the image of the main data and the image of the dependent data are displayed without overlapping each other. The information can be displayed without protruding from the screen, and thereby accurate and smooth chat communication can be performed.
[0009]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows an embodiment of a communication system provided with an image display control device according to the present invention. In the figure, client terminals T1 and T2 are connected to a server SV via a network NW.
[0010]
Each of the client terminals T1 and T2 includes a key input unit 20, a transmission processing unit 21, a reception processing unit 22, and a display unit 23. Further, a camera 24 is connected to the client terminal T1.
The key input unit 20 is composed of, for example, a keyboard, and is used for inputting information desired by the user. The information input by the key input unit 20 is input to the transmission processing unit 21 as text data. Further, image data of a still image or a moving image captured by the camera 24 is input to the transmission processing unit 21 via an external interface (not shown) of the client terminal T1.
The transmission processing unit 21 encodes and compresses the input text data and image data according to a predetermined encoding method, and then transmits the data to the network NW toward the server SV.
The reception processing unit 22 receives the compressed combined image data transmitted from the server SV via the network NW. Then, the original composite image data is reproduced by decoding the compressed composite image data, and the reproduced composite image data is supplied to the display unit 23 for display.
[0011]
On the other hand, the server SV includes reception processing units 10 and 16, cutout processing units 11 and 17, image position determination processing units 12 and 18, position adjustment processing units 13 and 19, a synthesis processing unit 14, a transmission processing unit 15 is provided.
[0012]
The reception processing unit 10 receives compressed image data as main data and text data as subordinate data transmitted from the client terminal T1 via the network NW. The received compressed image data is decoded to reproduce the original image data, and the reproduced image data is input to the cutout processing unit 11. On the other hand, text data is directly input to the image position determination processing unit 12.
[0013]
The cutout processing unit 11 identifies a background image and an object image from the image data input from the reception processing unit 10, and performs a process of cutting out data of the object image. Then, only the extracted object image data is input to the image position determination processing unit 12.
[0014]
The image position determination processing unit 12 determines a position when displaying the object image data on a display screen. The image position determination processing unit 12 generates a balloon image representing the content of the text from the text data input from the reception processing unit 10, and determines a position to arrange the balloon image near the object image. I do.
[0015]
The position adjustment processing unit 13 monitors the display positions of the object image and the balloon image whose display positions are determined by the image position determination processing unit 12, and determines whether the balloon image does not protrude from the display screen or overlaps with the object image. It is determined whether or not there is any overlap with the balloon image representing the content of the object image and text data sent from the other client terminal T2.
[0016]
If the above two conditions are not met, the information of the arrangement determined by the image position determination processing unit 12 is input to the combination processing unit 14 together with the object image data and the balloon image data. On the other hand, when the balloon image protrudes from the display screen, the position of the balloon image is adjusted so that the entire balloon image is close to the object image and within the screen, and when the balloon images overlap each other, the position of the object image is adjusted. Adjust so that it is close to the position and does not overlap the balloon image. Then, the adjusted arrangement information is input to the combination processing unit 14 together with the object image data and the balloon image data.
[0017]
Note that the reception processing unit 16, the cutout processing unit 17, the image position determination processing unit 18, and the position adjustment processing unit 19 are provided corresponding to the client terminal T2, and are provided corresponding to the client terminal T1. The same processing as that of the reception processing unit 10, the cutout processing unit 11, the image position determination processing unit 12, and the position adjustment processing unit 13 is performed.
[0018]
The synthesis processing unit 14 synthesizes the object image data input from the position adjustment processing units 13 and 19 and the balloon image data. At this time, the arrangement positions of the object image data and the balloon image data on the display screen are controlled according to the arrangement information set by the image position determination processing units 12 and 18 or the arrangement information adjusted by the position adjustment processing units 13 and 19. You.
[0019]
The transmission processing unit 15 compresses the synthesized image generated by the synthesis processing unit 14 according to a predetermined compression encoding method, and then transmits the compressed image to the network NW to the client terminals T1 and T2 that are communicating.
[0020]
Next, the operation of the server SV configured as described above will be described.
FIG. 2 is a flowchart showing the procedure and the contents of the image composition control.
When the client terminals T1 and T2 are connected to the server SV via the network NW, the server SV monitors the arrival of image data from the client terminals T1 and T2 in step 2a. In this state, when, for example, image data obtained by photographing the face of the user is received, the process proceeds to step 2b to read the received image data. Then, the server SV proceeds to step 2c, cuts out, for example, only the face portion of the object image data from the received image data, and sets a position for arranging the cut out object image data on the display screen in step 2d. decide.
[0021]
Next, the server SV monitors the arrival of text data from the client terminals T1 and T2 in step 2e. In this state, it is assumed that the user inputs the content of the transmission in the terminal devices T1 and T2, and that the text data of the transmission content is transmitted from the terminal devices T1 and T2. When the server SV detects the arrival of the text data in step 2e, the server SV proceeds to step 2f and reads the received text data. Then, the server SV proceeds to step 2g to generate balloon image data representing the content of the text data, and sets the display position of the generated balloon image on the display screen in step 2h. The display position of the balloon image is set so as to be close to the object image data of the corresponding terminal user received and cut out earlier.
[0022]
Next, the server SV proceeds to step 2i, and determines whether the balloon image does not protrude from the display screen or does not overlap an object image or a balloon image of another terminal user. As a result, if it is determined that none of the above applies, the process proceeds to step 2k to combine the object image data and the balloon image data.
[0023]
On the other hand, it is assumed that a part of the balloon image protrudes from the display screen or overlaps with an object image or a balloon image of another terminal user. In this case, the server SV determines that it is necessary to adjust the arrangement position of the balloon image, shifts to step 2j, where the balloon image does not protrude from the display screen and does not overlap with other object images and balloon images. Adjust to the position. At this time, the position adjustment of the balloon image is performed so as to maintain the relative position with respect to the object image of the same terminal user. In step 2k, the server SV combines the object image data and the balloon image data of each of the client terminals T1 and T2 whose display positions have been adjusted. The combined image data generated by this combining process is transmitted from the transmission processing unit 15 to the client terminals T1 and T2 that are communicating.
[0024]
FIG. 3 is a diagram illustrating an example of a process of generating synthesized image data by the above-described image synthesis control. In the figure, reference numeral 3a denotes image data coming from the client terminal T1, and object image data 3c, which is a face image of the user of the client terminal T1, is cut out from the image data 3a.
[0025]
Also, when text data representing the content of the transmission arrives from the client terminal T1, a balloon image 3d is generated based on the text data 3c. At this time, the text data 3c includes a text code and font information such as a font, a color, and a size. For example, in FIG.
"Taro (tomorrow morning, italic, size 16)", "Hi, (Ming, size 16)."
It is configured as follows. The server SV determines the font type and size of the character displayed in the balloon image according to the font information, but the font type and size of the character may be unified to a preset size.
[0026]
When the image data 3e and the text data 3g arrive from the client terminal T2, the object image 3f which is the face image of the terminal user is cut out from the image data 3e and the text data 3g as in the case of the client terminal T1. Is generated on the basis of. For example, as text data 3g
"How are you? Face image (gothic / bold, size 20)"
Is received, a balloon image as shown in FIG. 3 is generated according to the font information. That is, the information displayed in the balloon image is not limited to the character, but may be an image.
[0027]
The display positions of the object image 3b and the balloon image 3d of the client terminal T1 and the object image 33f and the balloon image 3h of the client terminal T2 are adjusted so that they fit within the display screen and do not overlap each other. The image is superimposed on a background image stored in advance. Then, the composite image data is transmitted to each of the client terminals T1 and T2 and displayed on the display screen. 3i of FIG. 3 shows a display example of the image data generated by the synthesizing process.
[0028]
Note that the balloon image is updated each time new text data arrives from the client terminals T1 and T2. The update processing of the object image is performed every time image data arrives from the client terminals T1 and T2 when the image data is a still image, and is performed every frame when the image data is a moving image. .
[0029]
By the way, the following various embodiments are conceivable for determining and adjusting the display positions of the object image and the balloon image on the display screen.
[0030]
(Example 1)
FIG. 4 is a diagram for explaining the position determination and adjustment processing of the balloon image.
[0031]
When the object image 4a is arranged below the display screen, the balloon image 4b is arranged above the object image 4a as shown in FIG. 4A. When the object image 4a is arranged on the upper side of the display screen, the balloon image 4b is arranged on the lower side of the object image 4a as shown in FIG. 4B. Similarly, when the object image 4a is arranged on the left side of the display screen, the balloon image 4b is on the right side of the object image 4a as shown in FIG. 4C, and the object image 4a is on the right side of the display screen. In this case, the balloon image 4b is arranged to the left of the object image 4a as shown in FIG. 4D.
Then, it is determined whether the balloon image 4b protrudes from the display screen or does not overlap the object image 4a. If the balloon image 4b protrudes or overlaps the object image 4a, the position of the balloon image 4b is further adjusted.
[0032]
By the way, when the above-described position determination processing is performed, the position of the balloon image 4b instantaneously moves between the upper side and the lower side of the screen or between the left side and the right side due to the change in the position of the object image 4a. Sometimes. In particular, when there is a strong restriction on the size of the display screen, the balloon image 4b may frequently reciprocate up, down, left, and right, and the screen may flicker. In this case, the balloon image 4b can be moved smoothly by setting an upper limit on the moving amount, moving speed or acceleration of the balloon image 4b.
[0033]
FIG. 5 is a diagram for explaining an example of the operation control. In this figure, it is assumed that the object image 5a has now moved from the position shown in FIG. 5A to a position overlapping below the balloon image 5b as shown in FIG. 5B. In this case, it is necessary to move the position of the balloon image 5b below the display screen. At this time, the amount of one movement of the balloon image 5b is kept within a certain range. By doing so, the movement of the balloon image 5b can be smoothed as shown in FIGS. 5C and 5D, and the flicker can be suppressed.
[0034]
Further, when the content of the balloon image changes, it becomes desirable to move the position of the balloon image. In this case, the position of the balloon image is moved and the shape of the balloon is changed. At that time, the change of the shape of the balloon is also given smoothness. For example, the changing speed or the acceleration of the length of the balloon in the vertical direction or the horizontal direction is limited, or the changing speed or the amount of change of the area of the balloon image is limited. By doing so, the shape of the balloon can be smoothly changed.
[0035]
FIGS. 6 and 7 show display examples when the shape of the balloon is changed, and when the length of the balloon in the vertical direction is changed, as shown in FIGS. 6 (a) to 6 (c). become. 7A and 7B when the length of the balloon in the horizontal direction is changed.
[0036]
Note that, even when the user moves the position of the balloon image by, for example, a manual operation and changes the display position of the object image accordingly, the change speed or the acceleration may be limited. This makes it possible to smoothly change the display of the object image.
[0037]
(Example 2)
When the object image moves, the balloon image moves accordingly. First, an example of the movement of the object image is shown in FIG. FIG. 8A shows a display screen of the client terminal T1, and FIG. 8B shows a display screen of the client terminal T2. In the initial state, it is assumed that the client terminal T1 displays the object image 8a of the user, and the client terminal T2 displays the object image 8g. In this state, the user at the client terminal T1 moves the mouse pointer to 8d, for example, and then moves the mouse pointer from 8e to 8f while holding the object image 8a using a means such as a click. Then, on the display screen of the client terminal T1, the display position of the object image also moves from 8b to 8c following the movement of the mouse pointer.
[0038]
On the other hand, at the client terminal T2, the display position of the object image moves from 8h to 8i in conjunction with the movement of the object image at the client terminal T1. In general, the range that can be photographed by a camera is limited, and conventionally, the position of an object image cannot be freely moved on a display screen. The display position of the object image can be freely moved on the display screen without being restricted by the video.
[0039]
The display position of the balloon image is also moved following the movement of the object image. For example, in FIG. 9A, it is assumed that the object image 9a and the balloon image 9c are displayed as a pair and the object image has moved from 9a to 9b. Then, following the movement of the object image, the balloon image also moves from 9c to 9d as shown in FIG. 9B. Note that, even when the position of the balloon image is moved, the object image moves following this movement.
[0040]
Further, the moving range of the object image and the balloon image may be the entire screen or may be limited. The limitation of the movement range can be set according to the content, or can be set for each terminal. It is also possible not to permit any movement. For example, when a composite image using a game screen is shared as a background image, the composite image can be provided without impeding the progress of the game by restricting the movement range.
[0041]
(Example 3)
It is also possible to automatically delete the balloon image. FIG. 10 is a diagram for explaining this. When the balloon image 10a is displayed as shown in FIG. 10A, unnecessary information can be deleted by deleting the balloon image 10a as shown in FIG. 10B after a predetermined time (for example, 15 seconds) has elapsed. Can provide an organized, more user-friendly screen.
[0042]
Further, depending on the size, shape, and display position of the object image and the balloon image, the balloon image may not be properly displayed even if the display position is adjusted. In such a case, the display of the balloon image can be deleted.
[0043]
(Example 4)
Unnecessary balloon images are arranged as follows. FIG. 11 is a diagram illustrating an example of the processing.
[0044]
In a situation where the balloon image 11b is displayed on the object image 11a as shown in FIG. 11A, if a new balloon image 11c is to be displayed as it is, the balloon images 11b and 11c are mutually connected as shown in FIG. 11B. Will overlap. In this case, as shown in FIG. 11C, when a new balloon image 11c is displayed, the old balloon image 11b is deleted. By doing so, an unnecessary balloon image can be arranged and an easy-to-view display screen can be provided.
[0045]
In the above description, the maximum number of composite image components is set to 2. For example, if the maximum number is set to 3, up to two balloon images can be displayed side by side. If the maximum number is 1, display of the balloon image is not permitted. Further, in addition to setting the maximum number of composite image components, it is also possible to set the upper limit of the number of displayed images for each of the object image and the balloon image.
[0046]
Further, when complicated processing is allowed, it is conceivable to arrange unnecessary balloon images by setting the maximum value of the area occupied by the bounding rectangle formed by the entire composite image component. This method is excellent in that an effective screen can be provided by simple processing by managing the upper limit of the number of display components.
[0047]
Further, the maximum number of composite display components can be set for each user. For example, in FIG. 12, the upper limit for the user 12a is set to 2, and the upper limit for the user 12c is set to 1. In this case, when the user 12a speaks, one balloon image is displayed, but no balloon image is displayed for the user 12c. By setting the maximum number of display components for each user in this way, for example, in a game in which the game is divided into two or more teams, it is possible to talk with friends of the same team, but with friends of another team, Only facial expressions can be read.
[0048]
Further, as shown in FIG. 11A, when the upper limit value of the composite image component is 2 and the time from transmission of the balloon image 11b to transmission of a new balloon image 11c is as short as 3 seconds. Does not delete the balloon image 11b as shown in FIG. 11C, but temporarily changes the upper limit value of the composite image component to 3, and displays the two balloon images side by side as shown in FIG. . By doing so, it is possible to compare the contents of the new and old speech balloons.
[0049]
Also, in a system in which information can be broadcast to multiple user terminals or information can be transmitted only to specific user terminals, transmission of information transmitted only to specific user terminals Record the number of times. If it is determined that the user of the client terminal T1 and the user of the client terminal T2 have frequent conversations based on the record of the number of transmissions, for example, the composite image configuration of the client terminal T2 at the client terminal T1 is determined. Automatically increase the upper limit of the element. In this way, information that is likely to be required by the user of the client terminal T1 can be provided for a longer time, and a more comfortable composite image can be provided to the user of the client terminal T1.
[0050]
(Example 5)
An effect can also be given to the display mode of the balloon image. For example, when a balloon image corresponding to the object image 14a is displayed as shown in FIG. 14A, the balloon image 14b is displayed in a semi-transparent state as shown in FIGS. 14B and 14C. Is displayed so that it becomes gradually clearer. Also, when deleting the balloon image, the balloon image is gradually erased while being translucent, as shown in, for example, FIGS. A similar effect can be applied to an object image.
[0051]
Also, a method called dissolve can be used. For example, an image is divided into blocks of 32 × 32 or the like, and each 32 × 32 block is divided into 16 8 × 8 blocks, and divided into 16 steps. A similar effect can be obtained by displaying one 8 × 8 block for each block to display the entire block.
[0052]
By giving the above-described effects to the display of the object image and the balloon image, it is possible to reduce a stimulus caused by a sudden change in the display given to the terminal user watching the display screen, and to provide a more comfortable display screen. it can.
[0053]
As another example of the display effect of a balloon image, there is a hidden character effect in which part or all of the text to be combined is replaced with a hidden character such as “OOOO” or “XXXX” or a blank character. For example, in a situation where four terminal users are playing a game in two groups of two, a user belonging to the first group instructs the terminal of a user belonging to the second group to apply a face-down effect. Then, since the user belonging to the second group is notified only that the user is in a secret conversation, information indicating that the other party is likely to think about something is transmitted, and the interest of the game is further increased. Further, for example, if the user of the client terminal T1 specifies to apply the hidden character effect to an unpleasant character string, the user need not look at the unpleasant character string.
[0054]
(Example 6)
It is also possible to set an upper limit on the size of the balloon image. The reason why the upper limit is set for the size of the balloon image is to prevent the balloon image 16a from protruding outside the display screen due to the combination of the images as shown in FIG. Therefore, it is necessary to limit the size of the balloon image according to the available screen size.
[0055]
In addition, if the balloon image covers the entire screen, other composite display components such as an object image cannot be displayed. Therefore, for example, when the upper limit of the horizontal length of the balloon image is w and the upper limit of the vertical length is h, the upper limit is divided into two types, and w is determined according to the position of the balloon image. The screen size is set so that it is equal to or smaller than the screen size and h is equal to or smaller than 1/2 of the screen size, or w is equal to or smaller than 1/2 of the screen size and h is equal to or smaller than the screen size. Here, the size of the display screen does not necessarily have to match the size of the physical device. For example, when displayed in a window, the size of the display screen is the size of the window.
[0056]
If the display screen is not rectangular, such as a circle, an appropriate method is used, for example, in which the area of the portion where the balloon image comes out of the display screen is reduced to a certain pixel or less. As another method for determining the upper limit of the balloon image, there is a method of setting the upper limit of the size according to the number of transmissions of the balloon image information by the user. Also, when the screen is shared between terminals that can communicate with each other, the number of times the terminal has received the balloon image information may be used instead of the number of times the balloon image information has been transmitted.
[0057]
On the other hand, characters in the balloon image may be omitted. If an upper limit is set for the size of the balloon image, or if the size of the balloon image is determined by the shape of the balloon image, the size of the balloon image may be limited. However, if the size of the balloon image is restricted as shown in FIG. 16B while the balloon image 16a is displayed as shown in FIG. 16A, for example, information that cannot be displayed occurs. In this case, as shown in a balloon image 16b in FIG. 16B, if a part of the information is omitted using an abbreviation symbol "..." or the like, it can be displayed as a balloon image. It is also possible to cope by displaying the information in a reduced size.
[0058]
In addition, information that is not displayed can be displayed by using the following method. For example, when the mouse pointer is moved over the balloon image 16b, the size of the balloon image is increased and displayed. With this configuration, the user can display information in the balloon image as needed, and it is possible to achieve both easy viewing of the screen and sharing of the balloon image information.
[0059]
(Example 7)
If the display size of the object image is large, or if there are a large number of object images and balloon images to be displayed, the content of the balloon image 17b is changed as shown in FIG. 17c.
[0060]
In this case, when the mouse pointer 17d is moved over the balloon image 17b as shown in FIG. 17B, the display priority of the balloon image 17b is moved to the top layer of the screen, so that the user's view is improved. The balloon information can be displayed. The management of the display priority is realized by preparing the display priority information for the composite display component and performing the composition according to the priority.
[0061]
Here, the display priority information can be determined by, for example, assigning a rank value and sorting the combined display components by the rank value, or managing the combined display components as a list to determine the rank at the time of combining. It should be noted that the rank value can be set to a larger value as the front or a smaller value as the front.
[0062]
When the balloon image is moved to the uppermost layer of the display screen, the display priority information is saved, and when the mouse pointer moves away from the balloon image 17b, the balloon image is moved to the original lower layer position. You can put it back. Also, it is possible to give an effect of changing the frame of the balloon image 17b or the color in the frame only when the mouse pointer is located. This makes it easier to understand that the display priority of the balloon image 17b has been changed by the mouse pointer. Further, the effect of changing the frame of the speech balloon image or the color in the frame allows the user to select a desired composite display component even if the device can only input discrete information such as a push button.
[0063]
(Example 8)
When a plurality of balloon images are selectively displayed, if the display order is not managed, for example, as shown in FIG. 18A, a balloon image 18a displaying an old utterance is replaced with a balloon image 18b displaying a new utterance. It may be hidden.
[0064]
Therefore, when it is determined that the balloon image 18b is displayed behind the balloon image 18a, the display priority is changed so that the balloon image 18b is displayed in front of the balloon image 18a as shown in FIG. . In this way, it is possible to avoid a problem that the balloon image of the new statement is hidden by the balloon image of the old statement.
[0065]
When the composite display component is managed as a list, the component displayed on the uppermost layer of the normal display screen is the first or last component of the list. For this reason, the display priority of each component can be easily set by appropriately setting the order of registration in the list. Also, in the case where the display priority is managed by the display order value, for example, when a higher value is set as the uppermost layer, the current display priority value is changed to a value slightly larger than the maximum value, for example, by a value larger by one. realizable. Also, the reception time can be stored for each composite display component in addition to the display priority order.
[0066]
(Example 9)
The effect on the composite display component will be described with reference to FIG.
For example, when it is desired to make the object image 19a representing the winner of the game stand out specially, the display mode is changed so that only the object image 19a is brightened and the periphery is darkened as shown in FIG. By doing so, it is possible to obtain an effect as if a spotlight was applied to the object image 19a. Note that the object image 19b may be made more prominent by further increasing the brightness of the object image 19a without changing the brightness around the object image 19a.
[0067]
The image can be made more conspicuous by superimposing and combining a plurality of single colors while changing the mixing ratio. Further, the object image can be made more conspicuous by a pattern composed of a plurality of colors, for example, a rainbow pattern in which the colors are changed in a smooth gradation for each line.
[0068]
Further, for example, when the user corresponding to the object image 19a shown in FIG. 19A starts input operation of the keyboard, and it can be predicted that the display of the balloon image is performed, the same method is used. The image 19a may be made to stand out. By making the object image predicted to display the balloon image stand out in this way, it is possible to direct the user's line of sight to the object image before the balloon image is displayed. For this reason, the information displayed in the balloon image can be read more quickly, whereby the response information can be transmitted more quickly, and as a result, more comfortable communication becomes possible.
[0069]
(Example 10)
In general, text data, which is main information displayed as a balloon image, is transmitted and received in units of lines or the like. For this reason, a conversation is likely to be delayed.
[0070]
Therefore, in the present invention, when the user corresponding to the object image 20a starts inputting text data as shown in FIG. 20, first, a mark indicating that input is being performed, such as the cursor 20c, is displayed on the corresponding balloon image 20b. Combine and display. Subsequently, as the input of the text data progresses, the display position of the cursor is shifted as shown in FIG. As a result, editing delay is significantly improved, and more rapid grasp of information and more comfortable communication become possible.
[0071]
Even if the above example is carried out as it is, it is difficult to know whether the editing is being performed or the input content is finalized. Therefore, for example, by adopting a method such as changing the color of the character being edited or changing the color of the frame or the frame of the balloon image, it is easy to see that the character being edited is undetermined. I do.
[0072]
(Example 11)
An embodiment of history management of a balloon image will be described with reference to FIG.
FIG. 21A shows a situation where the latest balloon image 21b is displayed. In this state, for example, the mouse pointer 21c is positioned on the balloon image 21b and clicked. Then, as shown in FIG. 21B, the past balloon image 21d is read from the memory and displayed. In this manner, the balloon image of the reception timing required by the user can be easily and completely displayed again.
[0073]
Also, as shown in FIG. 21 (b), when the image at the previous reception timing balloon is displayed and the image is clicked, the balloon image information at the previous reception timing is read out from the memory. Is displayed. That is, each time the button is clicked, the past balloon images are read out and displayed in the order of the newest reception timing.
[0074]
(Example 12)
An example of changing the shape of a balloon image will be described with reference to FIG. In a state where the object image 22a and the balloon image 22b are displayed as shown in FIG. 22 (a), when the transmitting user or the receiving user selects a desired shape from a balloon image shape list prepared in advance, for example, The balloon image 22b of the selected shape is displayed on the display screen. FIG. 22B shows an example of the shape of the balloon image. Further, the change of the color is performed in the same manner. For example, when the change of the frame color is instructed, the color of the frame is changed, and when the change of the background color in the balloon image is instructed, the background color in the balloon image is changed.
[0075]
By giving these effects to the balloon image, the user can select the balloon image having a desired shape, as well as change the shape of the balloon image according to the mood and emotion at that time. , And richer communication can be realized.
[0076]
(Example 13)
An example in which a composite image is displayed without using a balloon image will be described with reference to FIG.
[0077]
As shown in FIG. 23, in synthesizing an image, text data 23a to be displayed in a balloon image is synthesized and displayed with an object image without using a balloon image. In this way, the area occupied by the balloon image can be used as a display area for another image such as an object image. This display method is particularly effective in a device having a limited display area such as a portable terminal.
[0078]
In the above description, the reduced information 23b for identifying the transmitting user is combined and displayed at the head of the text data 23a. However, identification information such as a name and a number expressed in the text data is combined and displayed. You may do so. Further, it is not always necessary to combine and display information for identifying the transmitting user. For example, when two users share a display screen and have a conversation, it is easier to view the display screen if the information identifying the transmitting user is not combined.
[0079]
(Example 14)
An embodiment will be described in which the clipping state of the object image is displayed in another window. FIG. 24 is a diagram used for the description. An image obtained by combining an object image 24a cut out from the image data sent from the client terminal T1 and an object image 24b cut out from the image data sent from the client terminal T2 is shown in FIG. Is shown. Another window 24c is formed in the composite image, and the entire image data before cutting out the object image 24b is reduced and displayed in the window 24c.
[0080]
By combining and displaying the entire image data before clipping while displaying the clipped object image 24a in this way, the user can recognize how the clipping process was performed. Then, when the clipping is not properly performed, it becomes a clue to grasp the cause.
[0081]
Note that the window 24c may be colored and displayed, and instead of coloring, by mixing a plurality of colors in a translucent state, the cut-out area and the remaining area are simultaneously displayed on one display screen. It becomes possible.
[0082]
(Example 15)
When the display size of the object image 25a is significantly smaller than the size of the display screen as shown in FIG. 25A, for example, transmission such as an image of another person who is accidentally present in the imaging area of the camera or accidentally passed by is performed. In many cases, the image is not a regular image of the user. When a small area is cut out as an object image as shown in FIG. 25B, the cut out image is not an expected object image, but is incorrectly extracted due to the influence of input device uncertainty such as noise. Probably.
[0083]
Therefore, it is determined whether or not the display size of the image cut out as the object image is equal to or larger than a preset size. Then, the cut-out object image is used for synthesis when the size is equal to or larger than the set size, and is not used for synthesis at other times, so that a problem in which an unexpected image is synthesized can be avoided. Here, a numerical value such as an area of 32 pixels or more can be used as a reference for the set size. By taking such measures, it is possible to prevent a balloon image from being displayed in association with the erroneously extracted object image.
[0084]
(Example 16)
As described above, the balloon image may not be displayed in an appropriate form depending on the information amount of the text data, the size of the object images to be displayed side by side, and the like.
[0085]
Therefore, it is determined whether or not the display of the balloon image can be appropriately displayed. If the text data cannot be displayed properly, the text data is output as sound instead of being displayed as a balloon image. For example, when the balloon image can be displayed without protruding from the display screen and without overlapping with the object image, the balloon image is combined with the object image 26a and displayed as shown in FIG. On the other hand, if the balloon image cannot be displayed properly, the received text data is converted to voice data by voice synthesis technology without displaying the balloon image as shown in FIG. The data is output from the speaker.
[0086]
Such display processing is effective when the balloon image is synthesized and the empty area of the display screen is reduced, so that the balloon image is not desired to be displayed. Also, when combining an object image and a balloon image of a plurality of transmission users, the balloon image of each transmission user is effective in a case where the size of the balloon image is so small that it is difficult to see.
[0087]
Furthermore, a plurality of types of voice quality of voice data may be prepared, and the voice quality or volume of voice data may be changed for each transmission user. By doing so, it is possible to output text data of a plurality of transmission users while distinguishing them. In addition, by adding a command for emphasizing a specific word in a sentence to the control information of the balloon image, for example, a prosodic emphasis can be added to the corresponding word in speech synthesis.
[0088]
Further, when outputting the text data of the transmission user by voice, the display position of the corresponding transmission user's object image on the display screen is determined, and the volume balance of the left and right speakers is variably controlled according to the display position. Is also good. For example, as shown in FIG. 27, when the object image of the transmitting user is displayed on the left side of the display screen, the synthesized voice data corresponding to the text data of the transmitting user is transmitted to the left speaker 27a more than the right speaker 27b. Is controlled and output so as to have a larger volume. By controlling in this way, a more realistic sound output can be realized.
[0089]
Here, the display position of the object image is determined, for example, by detecting which position on the display screen the horizontal coordinate position of the center of gravity of the object image is located. Note that calculating the center of gravity requires a large amount of calculation, so the center point of the left end and the right end of the smallest rectangle including the object image may be used instead. If the object image includes the face image of the user, the position of the second eye of the face image is detected and its center point is recognized, and the position of the center point is set as the reference position of the object image. Just use it.
[0090]
The control of the volume balance between the left and right speakers is performed as follows. For example, the reference position coordinates on the display screen are defined as -96 at the left end, 0 at the center, and 96 at the right end. Then, the center position of the object image is linearly converted into a numerical value based on the reference position coordinates, and the converted numerical value is associated with the gain of the volume. Then, the sound volume is changed by controlling the constant of the PCM conversion circuit or controlling the amplification factor of the analog audio circuit based on the gain.
[0091]
In the above description, the case where the text data is output as speech using the speech synthesis technique has been described as an example. However, the speech synthesis technique may not necessarily be used. For example, the speech data input to the microphone by the transmitting user at the client terminal may be used. May be output as it is.
[0092]
FIG. 28 illustrates an example of audio output control in a case where another home electric appliance having a speaker such as a television receiver exists around the client terminal. The client terminal monitors the presence / absence of audio output from the speakers 28e and 28f of the terminal itself, and detects a period during which the audio output is performed, and lowers the volume of the speakers 28b and 28c of the home electric appliance 28a during this period. In this way, the user performing the communication can clearly hear the voice of the communication partner user output from the speakers 28e and 28f of the client terminal.
[0093]
(Example 17)
Generally, there are various restrictions on the camera, and particularly when the installation position is restricted, for example, as shown in FIG. 29A, the object image 29a in the captured image data is displayed in a rolled state. Sometimes. Therefore, in such a case, the display direction of the object image is determined, and the direction of the object image is rotated by 90 degrees to the right as shown in FIG. Combine. At this time, the display position and the display direction of the balloon image are adjusted according to the display position and the display direction of the object image.
[0094]
This makes it possible to always display the object image in an upright state regardless of the installation state of the camera. As a result, even when the object images and the balloon images of a plurality of terminal users are displayed on one display screen, it is possible to clearly display the correspondence between the object image and the balloon image so as not to erroneously recognize the correspondence.
[0095]
Other methods for correcting the display direction of the object image include the following. That is, if the input video is synthesized as it is, if the user moves left due to the characteristics of the camera, the user in the synthesized image will move right. This gives the user a sense of discomfort. Then, by synthesizing the display position of the object image with the balloon image after inverting the display position of the object image, the movement of the synthesized image matches the user's sense, and the synthesized image can be viewed more comfortably. Also in this case, the display position of the balloon image is adjusted according to the display position of the object image.
[0096]
(Example 18)
In general, many image compression standards can handle only a limited shape such as a rectangle as an image shape. Therefore, an object image 30a having an arbitrary shape is synthesized with a background image having a basic shape conforming to the image compression standard and set to a specific color as shown in FIG. On the other hand, when reproducing the compressed image, if the specific color of the background image is removed as shown in FIG. 30B, the object image 30a having the above-mentioned arbitrary shape can be reproduced. Here, for example, a single color of green or blue is used as the specific color. Note that the specific color is not necessarily one color, and a plurality of colors such as green and blue may be used as the specific color. Also, if a standard that allows compression and reproduction of an arbitrary shape, such as the international standard MPEG4 (Moving Picture Experts Group 4), is adopted for compression and reproduction, an arbitrary shape can be used without using a specific color image. Compression and reproduction of an object image can be realized.
[0097]
(Example 19)
The server SV converts the user's voice input to the microphone at the client terminal into text data by performing voice recognition, and displays the converted text data 31b using a balloon image 31a. In this case, if the voice data of the user is converted to erroneous text data due to erroneous recognition, the user of the communication partner who is viewing the synthesized and displayed text data 31b cannot correctly understand the content of the transmission.
[0098]
Thus, for example, as shown in FIG. 31, a mark indicating that voice display is possible is generated, and this mark 31c is displayed together with the text data 31b by a balloon image 31a. Then, when the user designates and clicks on the displayed mark 31c with the mouse pointer, audio data corresponding to the text data 31b being displayed is reproduced and output from the speaker by the balloon image 31a.
[0099]
With this configuration, the user can listen to the transmission voice of the communication partner user as voice data as needed, and even if erroneous recognition of the voice data occurs as a result, the user will be able to hear the transmission content of the communication partner user. Can be correctly recognized. Further, even when voice data which is difficult to convert into text data, such as an animal squeal or a sound, is transmitted, the voice data can be reproduced by voice. Further, when the balloon image has such a configuration, the size of the balloon image can be made constant and small.
[0100]
As described in detail above, in this embodiment, an object image is cut out from the image data received from the client terminal T1, and a balloon image representing the content of the text is generated from the text data received from the client terminal T1, and the balloon image Is arranged near the object image. Then, when the balloon image protrudes from the display screen or overlaps with another balloon image or the like, the position of the balloon image is adjusted to a position not protruding from the display screen and not overlapping with another balloon image or the like. I have to.
[0101]
Therefore, according to this embodiment, when the balloon image protrudes from the display screen, the composite image is created after being adjusted to a position that does not protrude. Also, when a balloon image overlaps another balloon image or the like, a combined image is created after being adjusted to a position where it does not overlap. For this reason, all of the information displayed in the balloon image can be accurately displayed in an easily viewable state at all times.
[0102]
Note that the present invention is not limited to the above embodiment. For example, in the above embodiment, the case where the server SV creates a composite image after performing position adjustment has been described as an example. However, in the case of a system in which a client terminal is directly connected via a network NW and not via a server SV for communication, the client terminal may be provided with a function of adjusting the position of the image and a function of creating a composite image. Just fine.
[0103]
In this case, the configuration of the client terminal includes a case where the image data and the balloon image of the own user are transmitted and a position adjustment process and a composite image creation process of these images are performed, and a case where the image data and the balloon image are transmitted from another client terminal. When the images are received, it is conceivable that the position adjustment processing of these images and the creation processing of the composite image are performed.
[0104]
The object image representing the user is not limited to an image actually captured by the camera 24, and may be an animation image arbitrarily selected by the user or an image other than a person. In this case, the server SV allows the user to select the case where an arbitrary image sent from the client terminal is used as the user's object image, or to select an animation image or the like stored in advance in the storage unit of the server SV. It is conceivable that the used image is used as the user's object image.
[0105]
In addition, the image display control device according to the present invention is not limited to a client terminal or a server for performing chat communication via a network, but may be applied to, for example, a single personal computer or a workstation that creates game software or the like. It is possible.
[0106]
The gist of the present invention also relates to the type and configuration of the communication system, the configuration of the server and the client terminal, the procedures and contents of the image position adjustment processing and the image synthesis processing, the processing method of the image data, the display form of the balloon image, and the like. Various modifications can be made without departing from the scope of the present invention.
[0107]
【The invention's effect】
As described in detail above, according to the present invention, in the image display control device and the display control program provided in the server device or the terminal device interposed between the plurality of terminal devices, the adjusting means for adjusting the form of the composite image is provided. When the newly provided main data and subordinate data are combined to generate composite data that can be displayed on one display screen, the received main data image and subordinate data image are displayed in the display screen area. The combined form is adjusted so that the combined data does not overlap with each other and does not overlap each other, and the combined data adjusted in the combined form is transmitted to the terminal device of the communication partner and displayed.
[0108]
Therefore, according to the present invention, the image of the main data and the image of the subordinate data can always be displayed in an appropriate state without overlapping each other or protruding from the display screen, thereby enabling accurate and smooth communication. A control device and an image display control program can be provided.
[Brief description of the drawings]
FIG. 1 is a diagram showing an embodiment of a communication system in which an image display control device according to the present invention is provided.
FIG. 2 is a flowchart showing a procedure and contents of image synthesis control in a server SV shown in FIG. 1;
FIG. 3 is a diagram showing an example of a process of generating composite image data by image composition control in a server SV shown in FIG. 1;
FIG. 4 is a view for explaining position determination of a balloon image.
FIG. 5 is a view for explaining movement of a balloon image accompanying movement of an object image.
FIG. 6 is a diagram illustrating an example of a change in the shape of a balloon image.
FIG. 7 is a view for explaining another example of a change in the shape of a balloon image.
FIG. 8 is a diagram illustrating an example of movement of an object image.
FIG. 9 is a diagram illustrating a balloon image that moves following an object image.
FIG. 10 is a view for explaining automatic deletion of a balloon image.
FIG. 11 is an exemplary view for explaining arrangement of unnecessary balloon images;
FIG. 12 is a diagram illustrating a case where the maximum number of combined display components is set for each user.
FIG. 13 is a view for explaining a case where two balloon images are displayed side by side.
FIG. 14 is a diagram illustrating a display effect of a balloon image.
FIG. 15 is a diagram illustrating a display effect of a balloon image.
FIG. 16 is a diagram illustrating a case where an upper limit is set for the size of a balloon image.
FIG. 17 is a view for explaining the display order of balloon images.
FIG. 18 is a view for explaining display management of balloon image information.
FIG. 19 is a view for explaining effects given to a composite display component.
FIG. 20 is a view for explaining editing of balloon image information.
FIG. 21 is a view for explaining history management of a balloon image.
FIG. 22 is a diagram illustrating a change in the shape of a balloon image.
FIG. 23 is a view for explaining an example in which text data is displayed without using a balloon image.
FIG. 24 is a view for explaining display of an image clipping state.
FIG. 25 is a view for explaining a countermeasure for cutting out a small area not expected by the user;
FIG. 26 is a diagram illustrating the use of audio information instead of a balloon image.
FIG. 27 is a diagram illustrating a case where text is output as sound by a speech synthesis system.
FIG. 28 is a diagram illustrating volume control of an external device.
FIG. 29 is a view for explaining pre-processing when image data is input from a camera.
FIG. 30 is a diagram illustrating transmission and reception of a character image having an arbitrary shape via a network.
FIG. 31 is an exemplary view for explaining an example in which sound is converted to text and synthesized and displayed together with an audio / video file.
[Explanation of symbols]
SV… Server
NW… Network
T1, T2 ... terminal
10, 16 ... Reception processing unit
11, 17 ... cutout processing unit
12, 18 ... Image position determination processing unit
13, 19: Position adjustment processing unit
14 ... Synthesis processing unit
15: Transmission processing unit
20 ... Key input section
21 ... Transmission processing unit
22 ... Reception processing unit
23 Display unit
24 ... Camera

Claims (12)

An image display control device provided in the server device of the communication system for transmitting data including an image between the plurality of terminal devices via the server device, wherein the plurality of terminal devices are connected to the server device via the network.
Receiving means for receiving, via the network, main data and subordinate data related to the main data transmitted by the first terminal device;
Combining means for combining the main data and the dependent data received by the receiving means to generate combined data that can be displayed on one display screen;
In the synthesizing, adjusting means for adjusting the synthesizing mode so that the received image of the main data and the image of the dependent data are contained in the display screen area and do not overlap each other,
Transmitting means for transmitting and displaying the combined data, the combined form of which has been adjusted by the adjusting means, to a second terminal device, which is a communication partner of the first terminal device, via the network. Image display control device. In the image display control device provided in the terminal device of the communication system that directly connects a plurality of terminal devices via a network and transmits data including an image between the plurality of terminal devices,
Reception means for transmitting, via the network, main data and subordinate data related to the main data transmitted by the communication partner terminal device,
Combining means for combining the main data and the dependent data received by the receiving means to generate combined data that can be displayed on one display screen;
In the synthesizing, adjusting means for adjusting the synthesizing mode so that the received image of the main data and the image of the dependent data are contained in the display screen area and do not overlap each other,
An image display control device, comprising: display control means for displaying, on a display, the synthesized data whose synthesis mode has been adjusted by the adjustment means. In the image display control device provided in the terminal device of the communication system that directly connects a plurality of terminal devices via a network and transmits data including an image between the plurality of terminal devices,
Input means for inputting main data and subordinate data related to the main data, respectively;
Synthesizing means for synthesizing the main data and the subordinate data input by the input means to generate synthesized data which can be displayed on one display screen;
In the synthesizing, adjusting means for adjusting the synthesizing mode so that the received image of the main data and the image of the dependent data are contained in the display screen area and do not overlap each other,
An image display control device comprising: a transmitting unit that transmits the synthesized data, the synthesizing mode of which has been adjusted by the adjusting unit, to a terminal device of a communication partner via the network, and displays the synthesized data. When the synthesizing means synthesizes a plurality of pairs of main data and subordinate data respectively arriving from a plurality of terminal devices to generate synthesized data that can be displayed on one display screen,
The adjusting means adjusts a combination mode between the plurality of pairs of main data and subordinate data so that an image of main data and an image of subordinate data fall within a display screen area and do not overlap. The image display control device according to claim 1. It is determined whether or not images can be displayed without overlapping based on the adjusting operation of the combining mode by the adjusting means. If it is determined that the images cannot be displayed without overlapping, the first The image of the main data and the dependent data received at the second time after the first time is overlapped with the image of the main data and the dependent data received at the time 5. The image display control device according to claim 4, further comprising means for adjusting a composition mode. The adjusting means adjusts at least one of the position, size, and shape of the image of the main data and the image of the subordinate data so that the image of the main data and the image of the subordinate data do not overlap in the display screen area and do not overlap. The image display control device according to claim 1, wherein: The image processing apparatus according to claim 1, wherein the adjusting unit changes at least one of a position, a size, and a shape of the image of the dependent data according to the change of the image of the main data when the image of the main data changes. An image display control device according to any one of claims 1 to 3. When changing at least one of the position, the size, and the shape of the image of the dependent data in accordance with the change of the image of the main data, the adjusting means has an upper limit on at least one of the speed, acceleration, and the amount of the change. 8. The image display control device according to claim 7, wherein: Further comprising a storage means for storing the main data in advance;
When the combining means uses main data stored in the storage means as main data to be combined,
2. The image processing apparatus according to claim 1, wherein the adjusting unit adjusts the combining mode such that the image of the main data and the image of the dependent data read from the storage unit fall within a display screen area and do not overlap each other. 4. The image display control device according to any one of 1 to 3. An image display used in the server device of a communication system in which a plurality of terminal devices are connected to a server device having a computer via a network, and data including an image is transmitted between the plurality of terminal devices via the server device. A control program,
A process of receiving, via the network, main data and subordinate data related to the main data transmitted by the first terminal device;
Combining the received main data and the dependent data to generate combined data that can be displayed on one display screen;
During the synthesizing process, a process of adjusting the synthesizing mode so that the received image of the main data and the image of the subordinate data fall within the display screen area and do not overlap each other,
Transmitting the combined data adjusted in the combining mode to a second terminal device, which is a communication partner of the first terminal device, via the network and displaying the combined data;
An image display control program to be executed by the computer. In an image display control device provided in a terminal device that synthesizes and displays data including a plurality of images,
Input means for respectively inputting a plurality of main data and subordinate data related to each of the main data,
Synthesizing means for synthesizing the main data and the subordinate data input by the input means to generate synthesized data which can be displayed on one display screen;
In the combining, an adjusting unit that adjusts a combining mode so that the image of the main data and the image of the dependent data fall within a display screen area and do not overlap each other,
An image display control device, comprising: display control means for displaying, on a display, the synthesized data whose synthesis mode has been adjusted by the adjustment means. An image display control program used in a terminal device having a computer to synthesize and display data including a plurality of images,
A process of receiving a plurality of main data and dependent data associated with each of the main data,
A process of combining the received main data and dependent data to generate combined data that can be displayed on one display screen;
At the time of the synthesizing process, a process of adjusting a synthesizing mode so that the image of the main data and the image of the subordinate data fall within a display screen area and do not overlap each other
A process of displaying the synthesized data in which the synthesis mode has been adjusted,
An image display control program to be executed by the computer.

Images