JP2003101978A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a plurality of images at the same height (horizontal level) when displaying them on a screen in an image processor and to eliminate mental superiority and inferiority relations. SOLUTION: The image processor having an image input means 100, an image transmission means 101 and an image reception means 102 is provided with an input image positional information extraction means 103 extracting positional information of a reference part from an input image, a transmission image positional information extraction means 104 extracting the positional information of the reference part from a received image, a position comparison means 105 comparing the positional information of the reference part of the respective images in the case of displaying a plurality of the images, a display area moving means 106 moving a frame arrangement so as to display the reference part at the same height from a compared result, an image arrangement change means 107 changing image arrangement within a frame, an image size change means 108 changing an image size within the frame and a display area size change means 109 changing a frame size.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used in an image interactive apparatus such as a personal computer, a word processor, a workstation, a portable information tool, a mobile phone, a facsimile apparatus, a television apparatus, a television conference, and an image pickup apparatus. The present invention relates to an image processing device that transmits an image acquired by using a communication device via a communication line to a remote place and displays the received image using a display device. In particular,
A specific feature amount regarding the input image, for example, the eye position information in the human image is extracted, and based on the information,
The present invention relates to an image processing device capable of displaying an image on a screen in an arrangement suitable for image content.

[0002]

2. Description of the Related Art Conventionally, an image and voice of a person on the transmission side acquired by an image pickup device are transmitted to a communication partner via a communication line, while the image and voice of the person of the communication partner are communicated. There is known an image interactive device which receives via a line, displays the received image on a display device, and communicates with each other, and is disclosed in, for example, Japanese Patent Application Laid-Open No. 7-131764.

In the image interactive apparatus disclosed in this publication, the line-of-sight position of the person sending the communication is detected by the line-of-sight position detecting means, and the obtained line-of-sight position information and the image are communicated by the transmitting means. It is transmitted to the communication partner via the line. The image of the communication partner and the line-of-sight position information are received by the receiving unit via the communication line, and the line-of-sight position information of the other party received and the line-of-sight position detecting unit included in the own device are detected. From the line-of-sight position information, the line-of-sight position of the person image (communication partner) displayed on the display device,
It is configured to match the line-of-sight position of the interlocutor who is looking at the display device. According to this configuration, the position information of the line-of-sight of the interlocutors is transmitted to each other, and the eye position of the image of the communication partner is aligned with the position of the line-of-sight of the interlocutor who is looking at the display device. The position of the line of sight can be matched.

On the transmission side, the liquid crystal screen provided on the display device is made transparent, and the interlocutor is photographed from the front by a video camera arranged on the back of the liquid crystal screen. At the same time, infrared rays from the infrared light source arranged on the upper part of the liquid crystal screen are irradiated to and reflected by the eyes of the interlocutor, and which part of the liquid crystal screen is the line-of-sight position by the infrared image sensor arranged on the back surface of the liquid crystal screen. That information is obtained. Based on this information, how much the line-of-sight position is deviated with respect to the optical axis of the video camera is detected, and the line-of-sight position information is generated. Then, based on this line-of-sight position information, the write control circuit controls the write timing so that the line-of-sight position of the image matches the optical axis of the video camera, the image is stored in the image memory, and this is transmitted as communication data. To be done.

On the other hand, on the receiving side, the image in which the line of sight is aligned with the optical axis on the transmitting side is received and stored in the image memory, and at the same time, the line-of-sight position information is generated in the same manner as described above. Then, on the basis of the received line-of-sight position information of the other party and the line-of-sight position information of the interlocutor who is looking at the liquid crystal screen, the readout control circuit causes the position of the eyes of the interlocutor who is looking at the liquid crystal screen and the image of the interlocutor. The reading timing is controlled so that the timing of reading the eye position (the position of the other party's eye) matches, and the image is read from the image memory. As a result, an image in which the positions of the eyes of the interlocutor who is looking at the liquid crystal screen and the positions of the eyes of the image (the positions of the eyes of the other party) are matched is projected on the liquid crystal screen by the liquid crystal projector.

As described above, the transmitting side stores and transmits the image in which the line of sight is aligned with the optical axis, and the receiving side receives and stores the image in which the optical axis is aligned. The images corrected so that the positions of the eyes of the communication partner match are displayed, and the line-of-sight positions of the interlocutors can be matched.

[0007]

However, like the above-mentioned conventional image dialogue apparatus, by aligning the position of the line of sight on the display device, an apparatus which enables a natural dialogue while both parties face each other. Then, you can only match the lines of sight of two people facing each other.
When multiple person images of more than one person are displayed,
The image layout cannot be adjusted between the individual images. Therefore, when a plurality of person images are displayed, there is a problem that an observer who sees the display feels a superiority relationship between the person images. When a plurality of person images are displayed on the screen, depending on the arrangement of the images, for example, the person image displayed at a high position on the same screen is superior to the person image displayed at a lower position. I get the impression.

The present invention has been made in order to solve the problems of the prior art as described above, and when a plurality of images are displayed on the screen, the reference position of the image, such as eyes in a face image, is set. The image processing apparatus can adjust the image arrangement so that all the different parts are set to the same height, and prevent an observer from feeling a psychological superiority or inferiority when an observer views a plurality of images displayed on the screen. The purpose is to provide.

[0009]

An image processing apparatus according to the present invention includes an image input unit for inputting an image, an image transmitting unit for transmitting the image input by the image input unit, and an image transmitted from the outside. An image receiving means for receiving the image information, and an input image position information extracting means for extracting the position information of the part serving as the reference position of the image from the image input by the image input means,
From the image received by the image receiving means, a transmission image position information extracting means for extracting the position information of the portion serving as the reference position of the image, and when displaying a plurality of images on the screen of the image output means, each image From the position comparison means for comparing the position information of the respective parts serving as the reference position of the image and the comparison result of the position information by the position comparison part, the part serving as the reference position of each image is displayed on the screen of the image output means. Display area moving means for moving the position of the display area where each image is displayed so that the images are displayed at the same horizontal level, and image arranging means for arranging the image and the display area at the coordinates on the screen of the image output means, Display control means for outputting the image and display area arranged by the image arrangement means to the image output means,
Image output means for displaying the image supplied from the display control means in a display area is provided, thereby achieving the above object.

According to the above structure, the input image is transmitted to the interlocutor, and the positional information of the reference position such as the eyes in the case of a face image is transmitted from the input image and the transmitted image. By extracting and comparing the position information of the extracted input image with the position information of the transmission image and adjusting the arrangement of the frame (display area) for displaying the image and displaying the image, the screen is displayed. It is possible to eliminate the psychological superiority or inferiority relationship between the plurality of images displayed in.

An image layout changing means for changing the layout of the images in the display area may be provided so that the reference position of each image is displayed at the same horizontal level on the screen.

According to the above configuration, the position information of the extracted input image is compared with the position information of the transmitted image, the arrangement of the image in the frame is adjusted according to the comparison result, and the image is displayed. It is possible to eliminate the psychological superiority or inferiority relationship between the plurality of images displayed in.

Image size changing means for changing the size of the image in the display area may be provided so that the reference position of each image is displayed at the same horizontal level on the screen.

According to the above configuration, the position information of the extracted input image is compared with the position information of the transmission image, the size of the image in the frame is adjusted according to the comparison result, and the image is displayed. It is possible to eliminate the psychological superiority or inferiority relationship between the plurality of images displayed in.

A display area size changing means for changing the size of the display area may be provided so that the reference position of each image is displayed at the same horizontal level on the screen.

According to the above configuration, the position information of the extracted input image is compared with the position information of the transmitted image, and the size of the frame is adjusted based on the comparison result, thereby adjusting the size of the image in the frame. In this case, it is possible to prevent the problem that the image does not fit in the frame or the image does not fit in a good balance.

When displaying images so that the reference position of each image is displayed at the same horizontal level on the screen,
You may provide the display area fixing means which arrange | positions each display area on a screen in the same size, and arranges the adjacent frames at the same space | interval.

According to the above configuration, the position information of the extracted input image is compared with the position information of the transmission image, and when the image is displayed in the frame based on the comparison result, the size of the frame is kept constant and equalized. By adjusting the image size and image arrangement within the frame and displaying the image in the frame, multiple frames can be displayed on the screen in good balance, and multiple images displayed on the screen can be displayed. The psychological superiority and inferiority relationship can be eliminated.

A display in which each display area on the screen is arranged in a horizontal line with the reference position of each image as a reference so that the reference position of each image is displayed at the same horizontal level on the screen. Area alignment means may be provided.

According to the above configuration, the position information of the extracted input image and the position information of the transmission image are compared, and when the images are displayed in the frame based on the comparison result, a plurality of frames such as face images are displayed. In the case of, the plurality of frames can be displayed on the screen in a well-balanced manner by arranging them in a line with reference to the reference position such as the eyes, and at the same time, the psychological image of the plurality of images displayed on the screen can be displayed. The superiority or inferiority relationship can be eliminated.

When displaying images so that the reference position of each image is displayed at the same horizontal level on the screen,
Display area overlapping means for overlapping and displaying the display areas may be provided.

According to the above configuration, when the position information of the extracted input image is compared with the position information of the transmission image and the images are displayed using a plurality of frames according to the comparison result,
For example, by setting the frame that is being spoken, etc., a predetermined frame is displayed in the foreground, and the remaining frames are overlapped behind the frame displayed in the foreground to display a plurality of frames on the screen. In addition to being able to be displayed, the image considered most necessary at present can be displayed in a state where the psychological superiority or inferiority relation with other images is eliminated.

When displaying images so that the reference position of each image is displayed at the same horizontal level on the screen,
An image selecting means for selecting an arbitrary image and a selected image front display means for displaying the selected image on the foreground and displaying other images in an overlapping manner may be provided.

According to the above configuration, when the position information of the extracted input image and the position information of the transmitted image are compared and a plurality of frames are displayed in superimposition according to the comparison result,
The user can arbitrarily select the image to be displayed on the foreground, and by displaying the remaining frames behind the frame selected by the user, multiple frames can be displayed on the screen. In addition, the image desired by the user to be displayed in the foreground can be displayed in a state in which the psychological superiority or inferiority with other images is eliminated.

Even if the initial value of the position information of the part serving as the reference position of the image extracted from each of the input moving image and the received moving image is fixed and then the position information changes, An initial value fixing unit that does not change the image arrangement, the image size, and the display area may be provided.

When the position information of the extracted input image and the position information of the transmitted image are compared and the image is displayed according to the comparison result, the position information of the image once extracted is not always constant. . For example, when the face image moves, the position information may differ from the first extracted position information, but each time the position information is updated, the image layout, image size, and display area are adjusted according to the position information. If changes such as the above are frequently made, the screen becomes difficult for the user to see.

According to the above configuration, the image displayed by the once extracted position information is displayed thereafter without changing the image arrangement, the image size, the display area, etc., so that a screen display that is easy for the user to see can be obtained. Can be provided.

When there is a change in the position information of the part serving as the reference position of the image extracted from each of the input moving image and the transmitted moving image, the image arrangement and the image are adjusted according to the degree of change. In order to change the size and the display area, a change rate calculating means for calculating the change rate may be provided.

When the position information of the extracted input image is compared with the position information of the transmitted image and the image is displayed according to the comparison result, the position information of the image once extracted is not always constant. . For example, when the face image moves, the position information may differ from the first extracted position information, but each time the position information is updated, the image layout, image size, and display area are adjusted according to the position information. If changes such as the above are frequently made, the screen becomes difficult for the user to see. However, when a person in the image moves largely, it may be possible to obtain a more appropriate image by changing the image arrangement, the image size, the display area, and the like according to the movement of the person image.

According to the above configuration, once the extracted position information is stored, the change in the position information is calculated when the position information is updated, and when the degree of the change is larger than a predetermined numerical value. In addition, by changing the image layout, image size, display area, etc. and displaying it on the screen, the image according to the situation is output, and the psychological superiority or inferiority relationship between the plurality of images displayed on the screen is eliminated. can do.

A site selecting means for selecting a site serving as a reference position of an image from which position information is extracted may be provided.

According to the above configuration, when extracting position information of an arbitrary part from the input image and the transmitted image, the user can select a more appropriate part according to the situation. Thus, it is possible to eliminate the psychological superiority or inferiority relationship between the plurality of images displayed on the screen.

When displaying images such that the reference position of each image is displayed at the same horizontal level on the screen,
A display method selection means for selecting an arbitrary display method may be provided.

According to the above configuration, the position information of the extracted input image is compared with the position information of the transmission image, and when the image is displayed according to the comparison result, the user can select a display method suitable for the situation. Can be selected.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the arrangement of an image processing apparatus which is an embodiment of the present invention. This image processing apparatus has an image input means 100 for inputting an image, and a display output means 120 for dividing the screen into a plurality of display areas and displaying an image in each display area.

The image input means 100 is, for example, a scanner or the like, and inputs various types of images such as face images, portrait images, landscape images and the like. The image input from the image input unit 100 is supplied to the image transmission unit 101, and is transmitted by the image transmission unit 101 to a communication partner such as an interlocutor via a communication line or the like. The image input from the image input means 100 is processed by the input image position information extraction means 103.
The position information of the part serving as the reference position of the image is extracted as the coordinate value on the screen, for example.

The image receiving means 102 receives an image transmitted from a communication partner such as an interlocutor via a communication line or the like. From the transmission image received by the image receiving unit 102, the transmission image position information extracting unit 104 extracts the position information of the part which is the reference position of the image.

The position information of the part serving as the reference position extracted by the input image position information extracting means 103 and the position information of the part serving as the reference position extracted by the transmission image position information extracting means 104 are compared with each other by the position comparing means 105. Are compared with each other, and the difference between the position information is extracted.

The difference in the position information extracted by the position comparing means 105 is obtained by the display area moving means 106, the image layout changing means 107, the image size changing means 108, the display area size changing means 109, and the display area fixing means. 110
Display area aligning means 111 and display area overlapping means 11
2 and the image arranging means 118.

In the display area moving means 106, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. The frame position of the input image or the frame position (coordinates) of the transmission image is changed so that the reference position of the image and the transmission image have the same height (horizontal level) on the screen.

The display area moving means 105, for example, as shown in FIG. 7, sets two display areas on the screen of the display device, and when the transmission image and the input image are displayed side by side in each display area, In the case of a face image, the position (coordinates) of the frame in which each image is displayed is changed so that the eye positions are the same height A on the screen.

In the image layout changing means 107, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. The arrangement (coordinates) of the image with respect to the frame is changed within the frame in which any one of the images is displayed so that the reference position of the image and the transmission image have the same height on the screen.

In the image size changing means 108, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. Make sure that the reference position in the image and the transmitted image are at the same height on the screen.
Resize the image within the frame in which it is displayed.

The image resizing unit 108 changes the image layout by the image layout changing unit 107, and when a part of the image goes out of the frame as shown in FIG. The image is enlarged / reduced centering on the part to be set so that the image fits within the frame as shown in FIG.

In the display area size changing means 109, based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image, The size of the frame in which one of the images is displayed is changed so that the reference position of the input image and the transmission image have the same height on the screen.

The display area size changing unit 109 changes the image arrangement by the image arrangement changing unit 107.
For example, when a part of the image goes out of the frame as shown in FIG. 8, the frame is enlarged or reduced so that the image fits in the frame as shown in FIG.

In the display area fixing means 110, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. When displaying the image in a frame so that the reference position of the image and the transmission image have the same height on the screen, all the frames are displayed in the same size, and the intervals between the frames are the same. Line up with.

The display area fixing means 110 calculates, for example, a frame size effective for fitting all the frames on the screen from the size of the screen and the number of frames, determines the position and size of the frame, and determines this frame. The position and size of other frames are determined according to the arrangement. Then, with respect to the image in each frame, the position, the size, and the frame interval of each frame are aligned by performing enlargement / reduction, movement, and the like around the reference position, for example, in the example of the face image in FIG. And the height A of the reference position
The image is displayed in the frame in the same state. The display area fixing unit 110 does not need the difference information which is the processing result from the position comparing unit 105, but the processing result is supplied from the position comparing unit 105 so that the processing is performed as a trigger. ing.

In the display area aligning means 111, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. When displaying an image in a frame such that the reference position of the image and the transmission image are at the same height on the screen, the frames are arranged in a horizontal row. The display area alignment unit 111 does not need the difference information which is the processing result from the position comparison unit 105, but the processing result is supplied from the position comparison unit 105 so that the processing is performed as a trigger. ing.

The display area arranging means 111 is arranged to display a plurality of frames on the screen when a plurality of frames are arranged on the screen such that the reference position of each frame, for example, the eye height A in the example of the face image shown in FIG. Since the sizes are different, the size that all the images fit on the screen is calculated. Then, by enlarging / reducing each frame while maintaining the respective aspect ratios, all the frames are displayed on the screen.

In the display area superposing means 112, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. When displaying an image in a frame so that the reference position of the image and the transmission image are at the same height on the screen, only the predetermined frame is displayed in the foreground, and the remaining frames are displayed. Stack it behind it.

The display area superimposing means 112 is designated when displaying a plurality of frames on the screen at a portion which becomes a reference position for each frame, for example, the eye height A is aligned in the example of the face image of FIG. The frames are overlapped so that only the frames displayed are in the foreground and the remaining frames a and b are displayed behind the left frame. The display area overlapping means 112 does not need the difference information which is the processing result from the position comparing means 105, but the processing result is supplied from the position comparing means 105 so that the processing is performed as a trigger. ing.

In the image selecting means 113, the position comparing means 1
On the basis of the difference between the position information of the part serving as the reference position of the input image extracted by 05 and the position information of the part serving as the reference position of the transmission image, the part serving as the reference position between the input image and the transmission image is displayed on the screen. When the image is displayed in the frame so as to have the same height, the selection screen is displayed on the image output unit 120 via the display control unit 119 so that the user can select the frame displayed on the foreground. It is displayed and the frame displayed on the foreground is determined from the selection result of the user.

In the selected image front display means 114, the image selected by the image selection means 113 is displayed on the foreground, and the other frames are overlaid behind it.

In the initial value fixing means 115, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. When the frame display position is determined once when the image is displayed in the frame so that the reference position of the image and the transmission image are at the same height on the screen, the position information at that time is initialized. The value is fixed so that the image arrangement in the frame, the image size in the frame, the frame arrangement, the frame size, etc. are not changed.

For example, when the image to be displayed is a moving image or the like, the height of eyes or the like is not always constant in the case of a part serving as a reference position, for example, in the case of a face image, the frame once determined Depending on the arrangement, the frame size, the image arrangement within the frame, the image size within the frame, etc., the eyes cannot always be displayed at the same height on the screen. However, if the frame layout, frame size, image layout within a frame, image size within a frame, etc. are changed each time according to the movement of the image, the frame layout, the frame size, the image layout within the frame, the image layout within the frame, etc. The image size is not fixed and the screen is difficult for the user to see. To prevent this, once the frame arrangement, the frame size, the image arrangement within the frame, the image size within the frame, etc. are determined, the initial value fixing means 115 fixes the position information at that time as the initial value, and then Then, the frame arrangement, the frame size, the image arrangement in the frame, the image size in the frame, etc. are not changed.

In the change rate calculating means 116, an input is made based on the difference between the position information of the part serving as the reference position of the input image extracted by the position comparing means 105 and the position information of the part serving as the reference position of the transmission image. When displaying an image in a frame such that the reference position of the image and the transmission image are at the same height on the screen, once the frame arrangement, the frame size, the image arrangement in the frame, the image in the frame After the size and the like are determined, if there is a large change in the arrangement of the part serving as the reference position, the position comparison means 105 again compares the position information to obtain a difference, and based on the information, the frame arrangement, the frame size, The image arrangement within the frame, the image size within the frame, etc. are determined.

For example, when the image to be displayed is a moving image or the like, the height of the reference position, for example, in the case of a face image, the height of the eyes is not always constant. Depending on the arrangement, the frame size, the image arrangement within the frame, the image size within the frame, etc., the eyes cannot always be displayed at the same height on the screen. For example, if the frame arrangement once determined by the initial value fixing means 115 is fixed as it is, the frame arrangement can be changed even when there is a large change in the arrangement of the displayed image, the size of the image, or the like. Instead, the screen may be difficult for the user to see. To prevent this, after the position information (coordinate value) is once fixed as an initial value, the difference between the coordinate value of the part that is the reference position of the displayed image and the fixed coordinate value is set in advance. If the value exceeds the numerical value, the position information is compared again and the difference is calculated to determine the frame arrangement, the frame size, the image arrangement within the frame, the image size within the frame, and the like.

In the part selecting means 117, the part desired by the user is selected as the part to be the reference position of the image.
For example, the selection screen is displayed on the image output means 120 via the display control means 119 so that the user can select an arbitrary part.
By displaying the image on the screen and storing the part or place specified by the user with a pointing device as a target part for position comparison, an image is displayed on the screen so that the coordinates are at the same height. You can Information on the selected part is supplied to the image position extracting means 103, 104.

The image arranging means 118 arranges the frames and images on the screen based on the coordinates (coordinates on the screen) of each frame and each image determined by the region selecting means 117 from the display area moving means 106. Is determined and display data is generated. Display control means 119
Then, image data such as a data signal and a scanning signal is generated based on the frame and image display data arranged by the image arranging means 118 and is supplied to the image output means 120. The image output unit 120 is a display device such as a liquid crystal display device, and the image supplied from the display control unit 119 is displayed in the frame.

The display method selection means 121 supplies a signal to the display control means 119 as a thumbnail showing the image arrangement state obtained by the image arrangement means 118, and causes the image output means 120 to display it as a selection screen. The user can select a desired one from the displayed thumbnails. Note that the display of thumbnails may be one type or multiple types for one screen.

Next, the image processing procedure in the image processing apparatus of this embodiment will be described using the flowchart shown in FIG.

First, in step s1, an image is input from the image input means 100. Here, the following description will be made assuming that the user inputs his / her face image, but an animal image, a landscape image, etc. may be used in addition to the person image. Similar to the person image, the position of the head of the animal image, for example, the eyes can be used as the position reference. In the landscape image,
For example, in the case of a landscape image of a mountain or the like, it is possible to regard the portion having the largest y coordinate among the ridgelines separating the color plane as the summit and set this point as the reference position. Also, the user may be allowed to select any location.

In the next step s2, the image transmission means 10
1, the input image is transmitted to another terminal device having a communication unit. Further, the image receiving unit 102 receives image data from another terminal device and captures the image in its own terminal device.

At the next step s3, the input image position information extracting means 103 extracts, as position information, the eye height, which is the reference position for aligning the image heights, from the input images.

As a method for measuring the eye height, for example, the method disclosed in Japanese Patent Application Laid-Open No. 8-322796 can be used. In this method, the positions of the eyes are estimated by extracting edges from the image, and the positions of the black eyes are estimated from the image that has been subjected to shading processing, and the positions of the eyes in the face image are measured. By this method, the eye position in the face image is measured, and the eye position coordinates (x, y) when the input image is displayed within the frame on the screen are acquired. At this time,
If the position coordinates of the extracted eyes are different on the left and right, the average value of both coordinates is taken.

Here, although the description is made with the part whose coordinates are acquired as the reference position as the eye, it may be a part other than the eye, and the user can select an arbitrary part by the part selecting means 117. You can also

In the next step s4, the transmission image position information extraction means 104 extracts, as position information, the eye height, which is the reference position for aligning the image heights, from the transmitted image.

In the next step s5, the position comparison means 10
5, the y-coordinate information indicating the eye height extracted from the input image and the y-coordinate information indicating the eye height extracted from the transmission image are compared to obtain the difference between them. .

The input image, which is the face image of the user, is displayed next to the transmitted image on the screen, but the frame size, the arrangement of images displayed in the frame, the image size, etc. are predetermined. ing.

In the next step s6, the display area moving means 106 moves and adjusts the arrangement position of the frame, so that an image with uniform eye height can be displayed on the actual screen. The display area moving unit 106 adds the difference information obtained in step s5 to the y coordinate of the frame arrangement of the input image and moves the display position of the image frame by frame. As a result, the arrangement of the eyes in the image moves by the amount of the difference, and the transmitted image and the input image have the same y-coordinate position of the eyes. Although the difference information is added to the frame of the input image in the present embodiment, it may be added to the frame of the transmission image. Further, although the input image and the transmission image are arranged in the present embodiment, the input images or the transmission images may be arranged.

In the next step s7, the image arranging means 11
8, the transmitted image is arranged at the predetermined coordinates, and the input image is arranged at the coordinates moved by the display area moving means 106. Then, the display control means 119
The image and frame whose display position has been determined are output to the image output means 120, and the image is displayed in the frame by the image output means 120.

Next, another image processing procedure in the image processing apparatus of this embodiment will be described with reference to the flowchart shown in FIG. Here, the three images are the size of the frame in which the images are displayed, the y-coordinate of the frame arrangement,
An example of changing the display position and the image size of an image within a frame when displaying with the same intervals between frames will be described. Regarding the arrangement of the three images, the input image and the transmission image are arranged at the right end, but other arrangements are also possible.

From step s01 to step s05,
Steps s explained using the flowchart shown in FIG.
Since the processing is the same as the processing from 1 to step s5, the description is omitted here.

In step s06, the display area fixing means 1
10, the horizontal width per frame is determined based on the number of frames displayed on the screen and the screen size in the image output means 120. When determining the width, the interval between frames is set, and this is set to 10 pixels or the like in advance. Screen size to frame interval ×
The range obtained by subtracting (the number of frames-1) is set as the frame displayable range, and the width of the frame is determined so that the frames can be displayed in the same size within this range. Also, the length of the frame in the vertical direction is determined based on the aspect ratio of the transmitted image.

When the transmitted image and the input image are arranged in a fixed frame by enlarging or reducing them in accordance with the frame size, the photographing environments are different from each other. Will be different. Further, when the frame arrangement is fixed, the frame itself cannot be moved to adjust the eye height.

In the next step s07, the image placement changing means 107 changes the image placement within the frame, so that the eye heights in the image can be adjusted.
The image layout changing means 107 adds the numerical value of the y coordinate to the position information of the previously extracted transmission image with reference to the y coordinate that is the eye height of the input image. This y-coordinate becomes the eye height when it is actually displayed on the screen, and the image is displayed in the frame according to this height. In the present embodiment, the y-coordinate, which is the eye height of the input image, is used as the reference, but the transmission image may be used as the reference.

When the image layout changing means 107 changes the image layout, a part of the image may go out of the frame, or conversely, the image layout may be biased with respect to the frame.

In step s08, such a problem can be prevented by enlarging or reducing the size of the entire image in the frame by the image resizing means 108. The image resizing unit 108 determines whether or not the difference between the x coordinate of the right eye and the x coordinate of the left eye obtained by the transmission image position information extracting unit 104 falls within a predetermined numerical range for the frame, What% if not within the range
Calculate whether the size can be accommodated by reducing the size. In addition, when the size is small, on the contrary, when the size is small, what percentage should be reduced to obtain a sufficient size is calculated. Based on the calculation result, based on the coordinates that are the average value of both eyes,
Change the image size. Although the size of the transmission image is changed in this embodiment, the size of the input image may be changed.

In the next step s09, the image arranging means 1
The image is arranged at the display position determined by 18, and the image and the frame whose display position is determined are output to the image output unit 120 by the display control unit 119, and the image is displayed in the frame by the image output unit 120. .

Next, another image processing procedure in the image processing apparatus of this embodiment will be described with reference to the flowchart shown in FIG. Here, an example will be described in which, when three images are displayed, the frames are enlarged or reduced while keeping the aspect ratio of each frame so that it fits on the screen.

From step s11 to step s15,
Steps s explained using the flowchart shown in FIG.
Since the processing is the same as the processing from 1 to step s5, the description is omitted here.

In step s16, the image layout changing means 1
07, the arrangement of the image displayed in the frame is changed based on the eye height.

In the next step s17, the display area alignment means 111 measures whether or not all the frames for displaying the images are arranged in a horizontal row on the screen.
If they can be arranged, the frame size is not changed.
Calculate whether it should be reduced.

In the next step s18, the display area size changing means 109 changes the frame size in accordance with the reduction ratio calculated by the display area aligning means 111. At this time, the aspect ratio of the frame is not changed.

In the next step s19, the image arranging means 1
The image is arranged at the display position determined by 18, and the image and the frame whose display position is determined are output to the image output unit 120 by the display control unit 119, and the image is displayed in the frame by the image output unit 120. .

Next, another image processing procedure in the image processing apparatus of this embodiment will be described with reference to the flowchart shown in FIG. Here, an example will be described in which the frame of the image desired by the user is displayed on the foreground, and the frames of the remaining images are overlapped and displayed in the arrangement based on the eye height.

Steps s011 to s015 are the same as the steps s1 to s5 described with reference to the flowchart shown in FIG. 2, and therefore the description thereof is omitted here.

In step s016, the display area overlapping means 112 determines the layout hierarchy. Here, a total of two frames, that is, the image that has been input first from the image input unit 100 and the image that has been transmitted first among the transmission images received by the image receiving unit 102 are displayed in the foreground. , The remaining images are displayed behind the frontmost transmitted image in the order in which they are transmitted. The arrangement of the frames of each image at this time is determined by the display area moving means 106.

In the next step s017, the display area moving means 106 determines the arrangement, and the display area overlapping means 1
The image of the frame whose hierarchy is determined by 12 is arranged by the image arrangement means 118. Then, the display control unit 119 outputs the image and frame whose display position has been determined to the image output unit 120, and the image output unit 1
The image is displayed in the frame by 20.

At the next step s018, the image selecting means 113 causes the image output device 120 to perform step s0.
The image selection screen is output together with the image output in 17.

At the next step s019, the user confirms whether or not the image displayed on the foreground is the desired image. The image selection screen is provided with an OK button or the like, and if the image desired by the user is displayed,
The image is selected as the foreground display image by pressing the OK button or the like, and the process proceeds to step s021. If the image is not the one desired by the user, step s020
Proceed to.

In step s020, the image selection means 11
3, the images can be sequentially arranged and displayed on the front surface of the screen of the display output means 120 by the cursor key or the like. Then, when the image desired by the user is displayed on the foreground, an image different from the first one is selected as the foreground display image by pressing the OK button or the like, and the selected image foreground display means 114 makes a final decision. The information of the obtained data is supplied to the image arranging means 118, and step s
Proceed to 021.

At step s021, the image of the decided frame is arranged by the image arrangement means 118. Then, the display control unit 119 outputs the image and the frame whose display position has been determined to the image output unit 120, and the image output unit 120 displays the image in the frame.

Next, another image processing procedure in the image processing apparatus of this embodiment will be described with reference to the flowchart shown in FIG. Here, from the result of comparing the position information extracted from the input image and the transmission image, after determining the frame display position and fixing the arrangement, if there is a large change in the arrangement of the part serving as the reference position, again, An example will be described in which position information is compared and a difference is calculated to determine a frame display position, an image display position, and the like to display an image.

For example, an image in which one's own face image is input,
Consider a case in which when the transmitted image is displayed on the screen, the person who is the subject of the transmitted image moves, and the height of the face changes and the height of the eyes also changes. In such a case, the face arrangement that is fixed to the initial value cannot achieve the purpose of matching the eye height in each image on the screen. Therefore, in the present embodiment, this problem is solved by changing the frame layout according to the situation on the image display side.

Steps s101 to s104 are the same as the steps s1 to s4 described with reference to the flowchart shown in FIG. 2, and therefore the description thereof is omitted here.

At step s105, the initial value fixing means 1
The position information of the input image extracted by the input image position information extracting unit 103 and the position information of the transmission image extracted by the transmission image position information extracting unit are stored by 15 and set as an initial value.

Here, the numerical value set as the initial value is thereafter set to a value exceeding the numerical value obtained by adding the predetermined numerical value to the initial value or a value smaller than the value obtained by subtracting the predetermined numerical value from the initial value. Unless changed, it will not be changed. here,
Let α be a predetermined numerical value. In addition, the y coordinate of the input image is ny, and the y coordinate of the transmission image is dy.

The input image position information extracting means 103 and the transmission image position information extracting means 104 automatically extract the position information every 180 seconds. Further, the change rate calculating means 116 compares the y-coordinate of the newly extracted input image with the y-coordinate extracted first. And
ny-α> y-coordinate value of the newly extracted input image> n
If y + α, the y-coordinate value of the newly extracted input image is stored in the initial value fixing means 115 as a new initial value.

Similarly, the change rate calculating means 116
The y-coordinate of the newly extracted transmission image is compared with the y-coordinate first extracted. If dy-α> value of y-coordinate of newly extracted input image> dy + α, the value of y-coordinate of the newly extracted transmission image is stored in the initial value fixing means 115 as a new initial value.

When the initial value fixing means 115 updates the initial value of the position information of the input image or the transmission image, in step s106, the position information of the input image set as the initial value by the position comparing means 105, The coordinates for the image arrangement are calculated by comparing the initial value and the position information of the set transmission image. Here, the y-coordinate information representing the eye height extracted from the input image is compared with the y-coordinate information representing the eye height extracted from the transmission image,
Take the difference.

In the next step s107, the display area moving means 106 moves the display position of the image frame by frame by adding the difference information obtained by comparing the position information to the y coordinate of the frame arrangement of the input image. As a result, the arrangement of the eyes in the image is moved by the amount of the difference, and the y coordinates of the eyes of the transmission image and the input image are at the same position.

At the next step s108, the image arranging means 118 arranges the transmission image as it is at the predetermined coordinates, and the input image is arranged at the coordinates moved by the display area moving means 106.

In the next step s109, the display control means 119 outputs the image and frame whose display position has been determined to the image output means 120, and the image output means 12
0 to display the image in the frame.

At the next step s110, the input image position information extraction means 103 and the transmission image position information extraction means 10
Position information is automatically extracted every 180 seconds.

In the next step s111, the change rate calculating means 116 compares the y-coordinate of the newly extracted input image with the y-coordinate extracted first. Then, in step s112, if ny-α> value of y-coordinate of the newly extracted input image> ny + α (yes),
Returning to step s105, the y-coordinate value of the newly extracted input image is set as a new initial value, and the initial value fixing means 1 is set.
Store in 15.

Similarly, in step s111, the change rate calculating means 116 compares the y-coordinate of the newly extracted transmission image with the y-coordinate extracted first. Then, in step s112, if dy-α> y-coordinate value of the newly extracted transmission image> dy + α, the process returns to step s105, and the y-coordinate value of the newly extracted transmission image is set as a new initial value. Initial value fixing means 11
Store in 5.

In step s112, ny-α> value of y-coordinate of newly extracted input image> ny + α or dy-α> value of y-coordinate of newly extracted transmission image>
If not dy + α, in step s113,
Images are output with the same frame arrangement.

[0111]

As described in detail above, according to the image processing apparatus of the present invention as set forth in claim 1, the input image is transmitted to the interlocutor, and the input image and the transmitted image are combined. For example, in the case of a face image, the position information of an arbitrary part such as the eyes, which is the reference position, is extracted, the position information of the extracted input image is compared with the position information of the transmission image, and the image is determined by the comparison result. By adjusting the arrangement of the frame (display area) to be displayed and displaying the image, the psychological superiority or inferiority relationship between the plurality of images displayed on the screen can be eliminated.

According to the present invention described in claim 2, the position information of the extracted input image is compared with the position information of the transmission image, and the image arrangement is performed by adjusting the arrangement of the images in the frame according to the comparison result. By displaying, the psychological superiority relationship between the plurality of images displayed on the screen can be eliminated.

According to the third aspect of the present invention, the position information of the extracted input image is compared with the position information of the transmission image, and the size of the image in the frame is adjusted according to the comparison result to form the image. By displaying, the psychological superiority relationship between the plurality of images displayed on the screen can be eliminated.

According to the present invention described in claim 4, the position information of the extracted input image is compared with the position information of the transmission image, and the size of the frame is adjusted according to the comparison result. When the size of the image is adjusted, the problem that the image does not fit in the frame or the image does not fit in a well-balanced manner can be prevented.

According to the present invention described in claim 5, the position information of the extracted input image is compared with the position information of the transmission image, and when the image is displayed in the frame according to the comparison result, By adjusting the image size, image layout, etc. in the frame and displaying the image in a frame that has a uniform size, it is possible to display multiple frames on the screen in a well-balanced manner. It is possible to eliminate the psychological superiority or inferiority relationship between the plurality of images displayed in.

According to the present invention described in claim 6, when the position information of the extracted input image is compared with the position information of the transmission image, and a plurality of images are displayed in the frame according to the comparison result, The frame, for example, in the case of a face image, the eyes,
By arranging them in a line with the reference position as a reference, multiple frames can be displayed on the screen in good balance, and the psychological superiority or inferiority relationship between the multiple images displayed on the screen can be eliminated. it can.

According to the present invention described in claim 7, when the position information of the extracted input image and the position information of the transmitted image are compared, and the image is displayed using a plurality of frames according to the comparison result. , A predetermined frame is displayed in the foreground by setting the frame being spoken, and the remaining frames are displayed behind the frame displayed in the foreground, so that a plurality of frames are displayed on the screen. Can be displayed, and the image considered most necessary at present can be displayed in a state in which the psychological superiority or inferiority relationship with other images is eliminated.

According to the present invention of claim 8, the position information of the extracted input image and the position information of the transmitted image are compared, and when the images are displayed by superimposing a plurality of frames on the basis of the comparison result. , The user can arbitrarily select the image to be displayed on the foreground, and the remaining frames are displayed behind the frame selected by the user to display multiple frames on the screen. As well as
An image desired by the user to be displayed in the foreground can be displayed in a state in which the psychological superiority or inferiority with other images is eliminated.

When the position information of the extracted input image and the position information of the transmitted image are compared and the image is displayed according to the comparison result, the position information of the image once extracted is not always constant. . For example, when the face image moves, the position information may differ from the first extracted position information, but each time the position information is updated, the image layout, image size, and display area are adjusted according to the position information. If changes such as the above are frequently made, the screen becomes difficult for the user to see.

According to the present invention of claim 9, the image displayed by the position information once extracted is
By displaying without changing the image layout, image size, display area, etc., it is possible to provide a screen display that is easy for the user to see.

When the position information of the extracted input image is compared with the position information of the transmission image and the image is displayed according to the comparison result, the position information of the image once extracted is not always constant. . For example, when the face image moves, the position information may differ from the first extracted position information, but each time the position information is updated, the image layout, image size, and display area are adjusted according to the position information. If changes such as the above are frequently made, the screen becomes difficult for the user to see. However, when a person in the image moves largely, it may be possible to obtain a more appropriate image by changing the image arrangement, the image size, the display area, and the like according to the movement of the person image.

According to the present invention of claim 10, the position information once extracted is stored, and when the position information is updated, the change in the position information is calculated, and the degree of the change is predetermined. If the value is larger than the specified value, the image layout, image size, display area, etc. are changed and displayed on the screen to output the image according to the situation and the psychology of the multiple images displayed on the screen. Can be eliminated.

According to the eleventh aspect of the present invention, when the position information of an arbitrary part is extracted from the input image and the transmitted image, the user can select a more appropriate part according to the situation. Can be selected, and the psychological superiority or inferiority relationship between the plurality of images displayed on the screen can be eliminated.

According to the twelfth aspect, when the position information of the extracted input image is compared with the position information of the transmission image and the image is displayed by the comparison result, it is appropriate depending on the situation. The user can select any display method.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus that is an embodiment of the present invention.

FIG. 2 is a flowchart illustrating an image processing operation in the image processing apparatus according to the exemplary embodiment of the present invention.

FIG. 3 is a flowchart for explaining another image processing operation in the image processing apparatus according to the embodiment of the present invention.

FIG. 4 is a flowchart for explaining another image processing operation in the image processing apparatus according to the embodiment of the present invention.

FIG. 5 is a flowchart for explaining another image processing operation in the image processing apparatus according to the embodiment of the present invention.

FIG. 6 is a flowchart for explaining another image processing operation in the image processing apparatus according to the embodiment of the present invention.

FIG. 7 is a diagram for explaining a display result when the display area is moved.

FIG. 8 is a diagram for explaining a display result when the image layout is changed.

FIG. 9 is a diagram for explaining a display result when the image size is changed.

FIG. 10 is a diagram for explaining a display result when the display area size is changed.

FIG. 11 is a diagram for explaining a display result when the size, the display height, and the interval of the display area are equalized.

FIG. 12 is a diagram for explaining a display result when the size of each display area is adjusted according to the screen area and all the parts serving as reference positions are displayed at the same height.

FIG. 13 is a diagram for explaining a display result when display areas are overlapped and displayed.

[Explanation of symbols] 100 image input means 101 image transmission means 102 image receiving means 103 input image position information extraction means 104 Transmission image position information extraction means 105 Position comparison means 106 display area moving means 107 image layout changing means 108 Image resizing means 109 display area size changing means 110 display area fixing means 111 display area alignment means 112 Display Area Overlap Means 113 image selection means 114 selection image front display means 115 means for fixing initial value 116 Change rate calculation means 117 site selection means 118 image arranging means 119 Display control means 120 image output means

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 5B057 CA08 CA12 CB08 CB12 CC01                       DA07 DA08 DA16 DB02 DC05                       DC36                 5C064 AA01 AA02 AC01 AC13 AC14                       AC15 AC22 AD06

Claims (12)

[Claims] 1. An image input unit for inputting an image, an image transmitting unit for transmitting the image input by the image input unit, an image receiving unit for receiving an image transmitted from the outside, and the image input unit. Input image position information extraction means for extracting position information of a part serving as a reference position of the image from the image input by, and position information of a part serving as a reference position of the image from the image received by the image receiving part. When transmitting image position information extracting means for extracting a plurality of images and displaying a plurality of images on the screen of the image output means,
Based on the position comparison means for comparing the position information of the respective parts serving as the reference positions of the images in each image, and the comparison result of the position information by the position comparison part, the part serving as the reference position of each image is displayed on the screen of the image output means. Display area moving means for moving the position of the display area in which each image is displayed so that they are displayed on the same horizontal level, and image arrangement means for arranging the image and the display area at the coordinates on the screen of the image output means. Display control means for outputting the image and display area arranged by the image arrangement means to the image output means, and image output means for displaying the image supplied from the display control means in the display area on the screen, An image processing apparatus equipped with. 2. The image layout changing means according to claim 1, further comprising image layout changing means for changing the layout of the images in the display area so that the reference position of each image is displayed at the same horizontal level on the screen. Image processing device. 3. The image size changing means for changing the size of an image in the display area so that a portion serving as a reference position of each image is displayed at the same horizontal level on the screen. Item 2. The image processing device according to item 2. 4. The display area size changing means for changing the size of the display area, so that the reference position of each image is displayed at the same height on the screen. The image processing device according to any one of 1. 5. When displaying images so that the reference position of each image is displayed at the same horizontal level on the screen, each display area on the screen has the same size and is between adjacent frames. The image processing apparatus according to any one of claims 1 to 3, further comprising a display area fixing unit that arranges the pixels at the same intervals. 6. The display areas on the screen are arranged in a horizontal row with respect to the reference position of each image so that the reference position of each image is displayed at the same horizontal level on the screen. The image processing apparatus according to any one of claims 1 to 5, further comprising: a display area aligning unit that controls the display area. 7. The display area overlaying means for overlaying the display areas when the images are displayed such that the reference position of each image is displayed at the same horizontal level on the screen. The image processing apparatus according to claim 4. 8. An image selecting means for selecting an arbitrary image when displaying the image so that a portion serving as a reference position of each image is displayed at the same horizontal level on the screen, and selecting the selected image as the maximum. The image processing apparatus according to any one of claims 1 to 4, further comprising: a selected image front display unit that displays the image on the front and displays another image in an overlapping manner. 9. A case in which an initial value of position information of a part serving as a reference position of an image, which is extracted from each of an input moving image and a received moving image, is fixed and then the position information changes. However, the image processing apparatus according to claim 1, further comprising an initial value fixing unit that does not change the image arrangement, the image size, and the display area. 10. When there is a change in position information of a part serving as a reference position of an image, which is extracted from each of the input moving image and the received moving image, image arrangement is performed according to the degree of change, The image processing apparatus according to claim 1, further comprising a change rate calculating unit that calculates a change rate for changing the image size and the display area. 11. The image processing apparatus according to claim 1, further comprising a site selecting unit that selects a site serving as a reference position of an image from which position information is extracted. 12. A display method selecting means for selecting an arbitrary display method when displaying an image so that a portion serving as a reference position of each image is displayed at the same horizontal level on the screen. The image processing apparatus according to claim 11.