JP2005092657A - Image display device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera-equipped mobile image display device/communication device and an imaging method that enable users to obtain 3D avatars as three-dimensional character images or their own images taken from the front, even if taking images while viewing a display part, the device being convenient for users to picture themselves. <P>SOLUTION: Misalignment is detected based on the positions/angles of the display part and the camera and on personal images taken with the camera, and 3D avatars, personal images and data are corrected and/or formed into images or data where the subject faces forward. The images or data are displayed and/or stored and transmitted. During photographing and when a subject of photography/text/image is displayed, control is effected in such a way as to perform an image process for the simultaneous display of images or text or the like and an automatic operation process, whereby the convenient camera-equipped mobile image display device/communication device and the imaging method can be achieved. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image display device and an image display method, for example, a portable image display device with a camera and a portable image display method with a camera.

  2. Description of the Related Art Conventionally, in an image display device with a camera, based on a photographed image, the face position, orientation, facial expression, and gesture are converted into a 3D avatar that is a 3D character and transmitted, or transmitted and stored as image data. A method of displaying has been proposed (see, for example, Patent Document 1).

  In addition, when photographing yourself while looking at the display unit with an image display device with a camera, etc., to obtain a front image, use a half mirror or the like to align the optical axes of the display unit and the camera. A technique for obtaining an image has been proposed (see, for example, Patent Document 2).

Japanese Patent Laid-Open No. 10-11609

JP-A-5-236470

  However, when a half mirror or the like is used, the system becomes large, so that there is a problem that it is not suitable for an image display device, particularly a portable image display device from the viewpoint of size reduction and weight reduction.

  In addition, when taking an image on the front of the image display device (especially when taking a self-portrait), it is necessary to shoot while looking at the display unit, so that the subject's line of sight is deviated from the center line of the camera, making it impossible to shoot in front there were. That is, in order to take a front image, it is necessary to look at the camera without looking at the display unit, or to arrange the display unit close to the center line of the camera to minimize the deviation from the front. If the front view can be taken while looking at the camera, the arrangement of the cameras is limited. This is a problem particularly in the case of an image display device in which component placement is performed in a limited space.

  Furthermore, in the case of a movable rotary camera or the like, there is a problem that the direction is shifted depending on the angle at the time of use because the image looks different depending on the rotation angle. In addition, recently, since the display unit is also movable, there is a problem in that it looks different depending on the positional relationship with the camera and the direction is shifted.

  Due to the above-described problems, if a 3D avatar is generated based on an image taken from an obliquely shifted position, an obliquely shifted 3D avatar is created. In such a case, it is appropriate to eliminate or reduce the deviation by correcting and generate a 3D avatar facing the front.

  Accordingly, an object of the present invention is to solve the above-described problems and provide an easy-to-use image display device.

  To achieve the above object, the present invention provides a camera that captures a subject and outputs image data of the subject, a control unit that generates a character image of the subject based on the image data, and a character image of the subject In the image display device including the display unit, the control unit detects a deviation between the direction in which the subject is facing and the direction in which the subject is facing from the image data, and the direction and the direction of the direction are detected. A character image in which the deviation is corrected is generated, and the display unit displays the character image corrected by the control unit.

  The present invention also provides a camera that captures a subject and outputs image data of the subject, a control unit that generates a character image of the subject based on the image data, and a display unit that displays the character image of the subject The control unit detects a deviation between the direction in which the subject is facing and the direction in which the subject is facing based on the positional relationship between the camera and the display unit, and the direction And a character image in which the deviation in the direction is corrected is generated, and the display unit displays the character image corrected by the control unit.

  In addition, the present invention provides a first camera for photographing a photographer himself as a subject, a second camera for photographing a traveling direction of the photographer, and an image of the photographer photographed by the first camera. And a display unit that displays an image in the traveling direction of the photographer photographed by the second camera on the same screen.

  In addition, the present invention provides a first camera for photographing a photographer himself as a subject, a second camera for photographing a traveling direction of the photographer, and an outline of the photographer photographed by the first camera. And a display unit that displays the image of the photographer in the direction of travel taken by the second camera on the same screen.

  The present invention further includes a camera that captures a subject and outputs image data of the subject, and a display unit that displays the image of the subject based on the image data. In the displayable image display device, when the display unit displays the subject image and the text on the same screen, the display unit displays the subject image and the background image other than the subject in a single color. It is characterized by.

  The present invention further includes a camera that captures a subject and outputs image data of the subject, and a display unit that displays the image of the subject based on the image data. In the displayable image display device, the color change of the text is performed by any one of a photographing time, a photographing time limit and the number of characters of the text, or a voice detection.

  The present invention further includes a camera that captures a subject and outputs image data of the subject, and a display unit that displays the image of the subject based on the image data. In the displayable image display device, the scrolling of the text is performed by any one of a photographing time, a photographing time limit and the number of characters of the text, or a voice detection.

  Further, the present invention provides an image display method for photographing a subject, outputting image data of the subject, generating a character image of the subject based on the image data, and displaying the character image of the subject. Detecting a deviation between the direction of the subject and the direction of the subject from the data, generating a character image in which the direction and the deviation in the direction are corrected, and displaying the corrected character image. Features.

  Further, the present invention provides an image display method for photographing a subject, outputting image data of the subject, generating a character image of the subject based on the image data, and displaying the character image of the subject. The direction of the subject and the direction of the subject are detected based on the angle between the direction of the central axis of the camera that captures the image and the front direction of the display unit that displays the character image, and the direction And generating a character image in which the deviation in the direction is corrected, and displaying the corrected character image.

  For example, based on the position and angle of the display unit and camera of a portable display device with a camera, and a person image taken by the camera, the 3D avatar and person image and data are corrected to a front-facing image and data.・ Create, display, save and send.

  In addition, when displaying the image or text of the person or the other party displayed on the display unit at the same time, the color is displayed lightly or in a single color or only the outline is displayed so that it is easy to see at the same time.

  A camera that captures the direction of travel when moving is prepared, and the user can check the direction of travel by displaying an image of the direction of travel on the display.

  Furthermore, the scrolling of the text displayed on the display unit is automatically scrolled according to the presence or absence of sound and the passage of time.

  According to the present invention, it is possible to provide an easy-to-use image display apparatus and image display method.

  First, a conventional method for creating and displaying a 3D avatar from an image will be briefly described.

  In the conventional method, the direction and movement of the face are detected from the captured image, and a 3D avatar is created and displayed based on the detected direction and movement.

  Specifically, first, the face, eyes, nose, mouth, and the like are detected from the captured image, and the direction and movement of the face are determined. The method of determining the face direction and movement by detecting the face, eyes, nose, and mouth from the image is to match the face with various directions of the face, pattern matching with the skin color, and after identifying the face area, Various methods such as a method for identifying the nose and mouth by color detection, etc., and a Hough transform for detecting human black eyes (for example, IEICE Transactions D-II Vol.J84-D-II No.10 pp.2190 -2200 (October 2001, etc.) is known, and the explanation is omitted here.

  Next, the 3D avatar is drawn according to this direction and movement. Image data of the 3D avatar is prepared in advance in the memory, and as shown in FIG. 9, there is data for creating an image called a skeleton made of a polyhedron and a texture pasted on the surface. A display image is created and drawn by changing the direction and shape of the skeleton and texture based on the face direction and movement data obtained as described above from the captured image data.

  The above is the conventional 3D avatar display method. According to the conventional display method, when an image taken from an oblique direction is displayed, the image remains oblique, and the displayed image is uncomfortable.

  Hereinafter, an image display device according to the present invention will be described with reference to the drawings.

  First, the outline of the present invention will be briefly described with reference to FIG. FIG. 2 shows an outline of the first embodiment of the present invention, where the subject A is photographed by the image display device with camera 100 and the photographed image is replaced with a 3D avatar and displayed on the display unit 2. It is a figure which shows the example in the case of having performed.

  When the camera 1 of the camera-equipped image display apparatus 100 in FIG. 2 tries to photograph the subject A facing the display unit 2, the obtained image data is an obliquely oriented image such as a. This is because the subject A is intended to be photographed while checking the display unit 2, and the line of sight of the subject A deviates from the center line of the camera. If a three-dimensional avatar is created as it is based on this image a, a three-dimensional avatar facing diagonally will be created. In the present invention, a three-dimensional avatar image b in which the direction is corrected based on the direction of the subject detected from the image a, the position / direction of the camera 1, the position / direction of the display unit 2, and the like is created. The image B is displayed on the display unit. By doing in this way, even if it image | photographs looking at the display part 2, the image B from the front can be obtained.

  By using the above method, the image and the position and direction of the camera and display unit can be processed even from an oblique image facing the display unit without facing the camera, and the angle of the image or data to be displayed, saved, or transmitted can be corrected. Images, data to be displayed / saved / transmitted can be made uncomfortable facing the front. The outline of the image display device and the image display method of the present invention is as described above.

  Details of the configuration and operation of the embodiment of the present invention will be described below in detail with reference to FIGS.

  First, the configuration of the image display apparatus according to the first embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is a configuration diagram of an image display apparatus with a camera according to a first embodiment of the present invention. An image display apparatus with a camera 100 shown in FIG. 1 includes a camera 1, a display unit 2, a control unit 3, a memory unit 4, an input / output unit 5, and a transmission / reception unit 9. The image recognition processing unit 6, the direction analysis unit 7, and the image conversion processing unit 8 are included therein.

  The camera-equipped image display device 100 is connected to the external network 200 via the transmission / reception unit 9 and is connected to other communication devices and the like.

  The camera-equipped image display device 100 is a communication terminal such as a camera-equipped mobile phone, and is a device that captures an image and stores and transmits the captured image.

  The camera 1 is an imaging device that can capture a still image or a moving image of a subject, and outputs image data of the captured subject. The display unit 2 displays information such as images and characters on a display screen such as an LCD, and displays an image of a subject, a three-dimensional character image thereof, and the like. The control unit 3 is, for example, a CPU, and controls the display unit 2, the memory unit 4, the input / output unit 5, the transmission / reception unit 9, and the like, and controls input / output and processing of information such as images and operation of the telephone. In addition, the control unit 3 includes an image recognition processing unit 6, a direction analysis unit 7, and an image conversion processing unit 8. Thus, the direction of the subject and the direction of the subject are determined from the image data of the subject. A direction shift is detected, and a three-dimensional character image in which the direction and the direction shift are corrected is generated. Such a three-dimensional character image is generated based on the image data of the subject and the data held in the memory unit 4. Here, the three-dimensional character image means an image in which a texture is pasted on a polyhedral skeleton as shown in FIG. Here, a three-dimensional character image will be described as an example, but not limited to three dimensions, a two-dimensional character image may be used. The memory unit 4 is, for example, an IC, and stores data and the like necessary for generating an image and a three-dimensional character image, and stores device-specific data. The input / output unit 5 performs an operation of inputting a user command or voice to the control unit 3 or outputting a voice, such as a voice input / output device such as a key button or a microphone / speaker. The image recognition processing unit 6 performs processing for extracting the position and direction of the subject's face, features such as eyes, nose, and mouth based on the image data obtained by the camera 1 and detects the direction in which the subject is facing. . The direction analysis unit 7 detects a shift in the direction in which the subject is facing based on the data of the image recognition processing unit 6. The image conversion processing unit 8 uses the data of the image recognition processing unit 6 and the direction analysis unit 7 to perform processing for creating an image such as a moving image or a still image to be displayed, and corrects the direction and direction deviation. Generate an image. The transmission / reception unit 9 may be a communication unit or a cable of a mobile phone, and transmits / receives data to / from the outside to transmit / receive image data, data necessary for image creation, data of a three-dimensional character image, and the like. The above is the system configuration of the first embodiment.

  Next, the operation of the camera-equipped image display apparatus 100 according to the first embodiment will be described with reference to FIGS.

  First, the face angle detection operation will be described with reference to FIG. FIG. 2 shows an outline of the first embodiment of the present invention, where the subject A is self-photographed by the image display device with camera 100, and the photographed image is replaced with a 3D avatar on the display unit 2. An example of display is shown.

  When the subject A facing the display unit 2 is photographed using the camera 1 of the camera-equipped image display device 100 of FIG. 2, the image is inclined as shown in the image a, and the image data is shifted in the oblique direction. Will be obtained. This is because the subject A is intended to be photographed while checking the display unit 2, and the line of sight of the subject A deviates from the center line of the camera. At this time, the angle in the oblique direction is determined by the distance from the display unit 2 to the subject A and the positional relationship between the camera 1 and the display unit 2. That is, if the camera 1 and the display unit 2 are separated from each other, the line of sight of the subject A is also shifted from the camera accordingly, and the angle increases. On the contrary, if the camera 1 and the display unit 2 are close, the angle decreases. In addition, the angle increases as the distance from the display unit 2 to the subject A decreases, while the angle decreases as the distance increases. It should be noted that in self-portrait where the user takes an image while holding the image display device himself, the angle tends to increase because the distance from the display unit 2 to the subject A is close, and correction is particularly necessary. is there.

  In order to correct this angle, it is necessary to detect the face, direction, and feature of the subject A from the image a as described above. This detection is performed by the image recognition processing unit 6 of the control unit 3. As described above, for example, a skin color region is extracted from an image, the vicinity thereof is determined as a face, and the position of the eyes, nose, and mouth of the face is detected from the region by pattern matching or color detection as described above. For eyes, black detection and Hough transform processing for detecting a circle can be used. For mouths, red detection can also be used, and position detection can be performed by various methods. Then, the eyes, nose, and mouth characteristics of the face are detected, and data or the like for creating the direction in which the face is facing or the facial expression is obtained.

  As described above, the face angle is detected.

  Next, the operation for correcting the direction of the 3D avatar will be described with reference to FIGS.

  First, the direction analysis unit 7 determines the front direction of the image to be displayed from the data such as the face direction obtained by the image recognition processing unit 6. When the face direction data obtained by the image recognition processing unit 6 is represented by an angle from the front direction of the camera, the X angle and Y angle values are obtained as shown in FIG. Here, the X angle and the Y angle mean a horizontal angle shift and a vertical angle shift, respectively, as shown in the coordinate axes of FIG. For example, in FIG. 3, the X angle of 45 ° indicates that it is shifted 45 ° to the right in the horizontal direction when viewed from the front of the camera, and this angle is an angle to be corrected in the horizontal direction. Similarly, the Y angle of 2 ° indicates that the angle is shifted 2 ° upward in the vertical direction when viewed from the front direction of the camera, and this angle is an angle to be corrected in the vertical direction.

  Here, since the numerical values of the X angle and the Y angle are detected every time, they are detected as the first time, the second time,... As shown in the table of FIG. The correction is performed based on the angle detected for each time. As shown in FIG. 4, the correction is made for the second and subsequent times based on the X angle and the Y angle in the thick frame line obtained for the first time. It is also possible to obtain the power correction X angle and the correction Y angle and perform correction using these angles. Here, as shown in FIG. 4, the correction X angle and the correction Y angle are the horizontal angle deviations based on the first detected X angle, and the first detected Y angle is used as a reference. It means the deviation of the angle in the vertical direction. For example, in the second detection result of FIG. 4, the correction X angle of −3 ° indicates a deviation between the first correction angle 45 ° and the second correction angle 42 °. The Y angle of 1 ° indicates a deviation between the first correction angle of 2 ° and the second correction angle of 3 °. That is, the direction analysis unit 7 corrects the direction of the display image based on this data. As shown in the table of FIG. 4, for example, based on the numerical values of the X and Y angles of the thick frame line obtained for the first time, for example. The corrected X angle and corrected Y angle for each time are obtained. Thus, by obtaining the correction X angle and the correction Y angle for the second and subsequent times on the basis of the angle of the first time, the front image when displaying as a front image such as a 3D avatar, that is, the face direction, The face direction of a 3D avatar or the like can be represented by a corrected X angle and a corrected Y angle. In this way, when the first angle is used as a reference, the processing time is shortened and high-speed display is possible. When the image and the image data are generated as described above, a front-facing image as shown in FIG. 4 b can be obtained. Can be sent through.

  Hereinafter, the flow of the above operation will be described with reference to the flowchart shown in FIG.

  The control unit 3 captures the captured image data from the camera 1 (S100), and the image recognition processing unit 6 detects the face angle from the image data (S101). Then, the direction analysis unit 7 estimates the first angle as the front angle (S102) and sets it as the standard angle. Further, the same direction analysis unit 7 corrects the face direction obtained from the image data and the face direction of the image such as the 3D avatar displayed from the standard angle (S103). According to the corrected direction, an image to be displayed such as a 3D avatar is created by the image conversion processing unit 8 (S104). The control unit 3 displays the created image on the display unit 2 (S105).

  The above is the flow of the operation of the correction method. In another example, in the above description, the first angle is estimated as the front angle and set as the standard angle. Then, an average of the angles may be obtained, and it may be estimated as the front angle to be a standard angle. In this way, even if the direction of the first direction is detected, even if it is turned in a strange direction, it is often statistically viewed from the front direction, so that correct correction can be performed, and the result is highly accurate. Can be obtained.

  As another example, as shown in FIG. 6, the estimation of the front angle (S102) is estimated from the mutual position, direction and angle of the display unit 2 and the camera 1 (S106), and the standard angle may be used. .

  When the display unit and the camera are fixed, the positional relationship between the display unit and the camera is unique to the image display device with a camera and does not change. Conversely, when the display unit and the camera are not fixed and movable, the positional relationship changes. Hereinafter, a description will be given with reference to FIG.

  As shown in FIG. 8, the point where the front direction of the display unit 2 and the central axis of the camera 1 intersect is less than one place, and the subject to be imaged is centered on the imaging while facing the front of the display unit 2. You must be near this point. This is because if it is displaced along the front direction of the display unit 2, it will deviate from the center of imaging, and if it is displaced along the central axis direction of the camera 1, the display unit 2 cannot be seen from the front.

  If the camera 1 and the display unit 2 are fixed and there is no change in the positional relationship between them, the angle data of the front direction of the display unit 2 and the central axis direction of the camera 1 is stored in the memory unit 4, and these The angle θ in the oblique direction of the subject photographed at one point connecting the two can be found. This may be used for angle correction as a standard angle.

  Even when the camera 1 and the display unit 2 are movable, such as a rotary type, and the positional relationship changes, the control unit 3 displays the changed data such as the position and angle of the camera 1 and the display unit 2 respectively. Or by detecting the mutual direction obtained from the set angle of the camera 1 and processing the mutual direction by the direction analyzing unit 7, the angle in the oblique direction of the subject can be estimated. Correction is possible by calculating. That is, in the control unit, the direction in which the subject is facing based on the positional relationship between the camera 1 and the display unit 2 (more specifically, the angle between the direction of the central axis of the camera 1 and the front direction of the display unit 2). Further, a deviation in the direction in which the subject is facing is detected, and a character image in which the direction and the deviation in the direction are corrected is generated.

  5 and FIG. 6 are combined to estimate the front angle (S102), detect the face angle from the image data (S101), and display unit 2 as shown in FIG. By using both of the angles estimated from the mutual position, direction, and angle (S106) of the camera 1, a more accurate standard angle can be obtained, and the correction accuracy can be increased.

  As described above, it is possible to obtain a correction angle for the deviation in the oblique direction of the captured image data. Based on this correction angle, the image conversion processing unit 8 in FIG. An image and image data to be stored in the memory unit 4 or transmitted to the outside via the transmission unit 9 are created.

  Next, a method for creating an image and image data will be described with reference to FIG.

  In the case of a 3D avatar, the memory unit 4 has the three-dimensional skeleton data in FIG. 9A and the texture data that is an image to be pasted on the skeleton surface in FIG. The memory 4 can also have regular expression image data, movement data, and the like.

  First, a 3D avatar skeleton is prepared, and the skeleton setting and movement and texture data of the image to be pasted are determined from the facial expression obtained from the facial feature data by the image recognition processing unit 6 to obtain a 3D avatar. Here, the corrected direction obtained by the direction analysis unit 7 is set as a viewpoint direction for viewing this 3D avatar, and image data for 3D display is created. Further, an image obtained from the viewpoint direction of viewing the 3D avatar is created by calculating the coordinate data of the skeleton points, the texture pasting angle, and the like. And the control part 3 can display the image which faced the front on the display part by outputting the image produced by correct | amending a direction to the display part 2. FIG. The control unit 3 can store images and 3D display image data in the memory unit 4, and can transmit images and 3D display image data via the transmission / reception unit 9.

  Here, the image is still image or moving image data, but the image data for 3D display is data for creating a 3D avatar. This data includes not only the above-described 3D skeleton data and texture data, but also movement data for moving the 3D avatar, display direction of the 3D avatar, data for facial expression deformation such as laughing and getting angry. In the case of a 3D avatar, only the motion data, the display direction, and the data for transforming the facial expression are transmitted to the communication partner, and the received motion data is prepared by preparing the 3D skeleton and texture on the communication partner side. 3D avatars can be displayed based on the above. In this case, since transmission can be performed with a small amount of communication data, it is particularly effective when the communication speed is low. The image data for 3D display to be transmitted is corrected and created by the image conversion processing unit 8, so that it is possible to send image data for 3D display that can be displayed corrected to the communication partner. .

  On the other hand, if an image or image data for 3D display is received from the outside via the transmission / reception unit 9, in the case of an image, the control unit stores the image in the memory unit 4 or displays it on the display unit 2. Or In the case of image data for 3D display, a 3D avatar image that is stored in the memory unit 4 or displayed in accordance with the image data for 3D display, including the 3D avatar skeleton and texture stored in advance in the memory unit 4 Created and displayed on the display unit 2. If all the elements for creating a 3D avatar image are included in the image data for 3D display, display can be performed using only the transmitted 3D display image data. However, in order to reduce the amount of communication data, There is also a method of receiving only motion data and using data stored in advance on the receiving side for other 3D avatar display data.

  The images to be transmitted in both sides of communication are 3D avatar images with corrected angles and 3D display image data, and 3D avatars can be used if images received or images created from image data are displayed on the display unit. The image displayed according to the present invention is free from a sense of incongruity with the angle corrected.

  In the above description, the angle correction of the 3D avatar image and the 3D display data using the 3D character is performed, but it is also possible to use the photographed person image. In other words, only a standard 3D avatar person's face skeleton is prepared, and texture data, which is an image to be pasted on the skeleton, is created from the photographed image, thereby correcting the angle using the photographed person image. It becomes possible to do. Specifically, the orientation of the face and the positions of eyes, nose, mouth, etc. are detected from the image taken by the above-described method, and the positions, eyes, nose, mouth, etc. of the human face skeleton for 3D avatar are detected. The position is associated and an image to be pasted on the skeleton is created from the corresponding position in the captured image. That is, in this case, a character image is generated by pasting a texture created from a captured image on the skeleton of the polyhedron shown in FIG.

  In the example of FIG. 1, since the image is from an oblique side, it is possible to easily paste an image of a visible part, but an invisible part is created with a symmetrical face. Further, if a plurality of cameras are used as shown in FIG. 10, the subject can be captured from various angles, so that an image can be created easily and accurately.

  As described above, the camera-equipped image display communication device 100 according to the first embodiment operates.

  Next, the configuration of the image display apparatus according to the second embodiment of the present invention will be described with reference to FIG.

  As is apparent from FIG. 11, the difference between the present embodiment and the first embodiment is that the transmission / reception unit 9 is replaced with the transmission unit 10. The operation is almost the same as that of the first embodiment, and the captured image and the image data for 3D display are stored or transmitted, and this is an image display device with a camera dedicated to transmission.

  Further, a configuration in which the transmission unit 10 in FIG. 11 is eliminated is also conceivable. In this case, the apparatus does not perform communication, and only captures and displays the corrected image.

  In addition, the direction of the 3D avatar is corrected in the above, but if a plurality of cameras are used, it is also possible to create a front image in which the direction is corrected directly from an oblique image captured by the camera. . Recently, it is called a stereo camera or the like, and is a technology used for stereoscopic recognition. If a plurality of cameras are used, the three-dimensional position information of the image can be obtained from the parallax of the plurality of images like the human eye using the principle of triangulation. The configuration is the same as in the first embodiment, and is the same as when a plurality of cameras 1 are used. The operation will be described with reference to FIG.

In FIG. 12, v1 and v2 are camera positions, and there are image 1 and image 2 obtained from each camera.
If it is known from the image processing of image 1 and image 2 that p1 and p2 correspond to each other, the three-dimensional position of the camera of v1 and v2 is known, and the three-dimensional position of point P is obtained as the intersection of the straight line v1p1 and the straight line v2p2. Can do. When taking a person, as described in the first embodiment, it is possible to detect the positions of eyes, nose, mouth, and the like from a single image. I understand the point. After that, calculation processing is performed as described above, and the three-dimensional position of each point of the person is obtained. Therefore, since the three-dimensional position of the subject can be known, a model like an avatar skeleton can be created. An image as a corresponding texture is extracted from a plurality of images obtained from the camera and pasted to the created skeleton. In this way, an avatar of a person who is not a character can be created from an image of the person who is the subject. Based on this, it is possible to create a front image in which the direction of the person is corrected by correcting the angle as in the first embodiment.

  As described above, an image and image data in which the direction of the subject to be photographed is corrected can be obtained, and the corrected image and image data can be displayed, stored, and transmitted. Even if a camera is not installed in the vicinity of the center of the display unit, it is possible to obtain a corrected image from the front during shooting while looking at the display unit. In particular, when the screen of the display unit becomes large, the camera is greatly disengaged from the central portion, which is effective in correcting this.

  The image display device / communication device with a camera and the photographing method of the present invention described so far can not obtain a single image because a plurality of cameras can be used to obtain images of subjects from various angles. Since an average of data from a part or a plurality of images can be obtained, more accurate image direction correction can be performed.

  Further, the direction correction of the image can be executed only at the time of self-portrait, and otherwise normal shooting can be performed. Some mobile phone cameras have a self-portrait camera and another camera. When a self-portrait camera is used, the direction of the image is corrected. When another camera is used, the direction is not corrected. Similarly, in the case of a rotary camera, when rotating the camera to the same side as the display unit and taking a picture, it is determined that it is self-portrait and direction correction is performed, and the camera is rotated to a different side from the display unit. If it is determined that it is not self-portrait, control is performed so as to stop the direction correction. Similarly, when the display unit rotates, direction correction is performed if the direction is the same as the camera, and direction correction is stopped if the direction is different. In the case of self-portrait, shooting is performed while looking at the display portion, so the direction correction is effective, but in other cases, the direction correction is not so necessary. By controlling in this way, if switching can be performed automatically, the setting work is saved and the usability is improved.

  Further, when the display unit and the camera are movable, such as rotating, the direction correction can be automatically stopped when the display unit is closed or the mutual position between the camera and the display unit is not clearly self-portrait. By detecting and calculating the rotation angle of the display unit and camera, the direction in which the display unit and the camera are facing can be determined, and if the self-portrait cannot be determined even when facing the same direction, the image direction is automatically corrected. Is controlled to stop.

  In addition, by controlling the image direction correction so that it can be stopped by setting, when using the image direction correction in the setting, it is suitable for self-shooting as described above, and when not using it, it is normal shooting. It is possible to shoot according to the user's intention. Setting can be performed by inputting a setting command from the user from the input / output unit 5 and storing the setting in the memory unit 4.

  Further, as described above, since the display unit can be seen at the time of self-photographing, the camera 1b is attached separately from the self-photographing camera 1a as shown in FIG. By displaying the image on the screen, the photographer can check the traveling direction while taking a picture. That is, a first camera for photographing the photographer himself and a second camera for photographing the traveling direction of the photographer, the photographer's image photographed by the first camera and the second camera are photographed. An image of the photographer's direction of travel is displayed on the display unit 2, so that attention can be paid to the direction of travel even when taking a selfie while walking. That is, if the user takes a picture while moving, it is dangerous because the direction of travel cannot be confirmed because the camera is seen. However, if this method is used, the direction of movement can be confirmed and safety is improved. When it is desired to confirm the position of self-photographing while viewing the image in the traveling direction, both the self-photographed contours can be displayed on the image in the traveling direction as shown in FIG. In other words, the contour image of the photographer photographed by the first camera and the image of the photographer traveling direction photographed by the second camera can be displayed on the display unit 2.

  Moreover, since the display part can be seen at the time of self-photographing, it is also conceivable to use the self-photographing while displaying text such as characters on the display part. At this time, I want to check my position while shooting while looking at the text. The captured image and text may be displayed simultaneously, but in this case, there is a problem that the text becomes difficult to see. Therefore, if the photographed person and the background are displayed as a shadow in a single color as shown in FIG. 14 and displayed together with the text, the characters of the text are easy to see and the user's own photographing position can be confirmed at the same time. That is, the subject image and the waste liquid image other than the subject can be displayed in a single color. In this case, only the outline of the person may be displayed as described above.

  In addition, when the subject's dialogue is displayed as text on the display, and you speak and shoot while watching it, you can change the text color on the display to indicate how far you have spoken, or text for long sentences It may be possible to scroll the display of. At this time, it is conceivable to operate the apparatus. However, when the apparatus is operated, there is a problem that an image captured by the camera is blurred. In particular, blurring becomes large in a small portable terminal. Therefore, character color change and scrolling may be performed automatically in time from the start of shooting. The speed at which the character color is changed or scrolled over time may be set by the user, or may be a predetermined speed. In addition, if there is a time limit for shooting, such as a camera-equipped mobile phone that can detect voice or automatically record a video, the time limit and the number of text characters are calculated and automatically performed within the time. Control is also conceivable. In order to automatically detect the sound and operate it, the control unit 3 determines the presence or absence of the sound input from the input / output unit in FIG. This can be achieved by stopping the operation. That is, text color change and text scrolling can be performed by any one of photographing time, photographing time limit and number of text characters, or voice detection.

  As described above, in a portable display device with a camera and a communication device using the same, a 3D avatar or a self-portrait front image can be displayed, stored, and transmitted while viewing a person, and even during shooting It is possible to display an easy-to-see display such as a shooting position, text, and the face of a communication partner on the display unit, and to realize a user-friendly shooting / display function and a device having the same. In addition, usability can be further improved by devising the simultaneous display of information on the display unit or by automating operations during shooting.

The figure which shows the system configuration | structure of the Example of this invention. The figure explaining the outline | summary of this invention. The figure explaining the operation | movement which detects an angle from the image of this invention. The figure explaining the operation | movement which performs angle correction | amendment from the image of this invention. FIG. 2 is a diagram (1) showing an operation sequence in the embodiment of the present invention. FIG. 2 is a diagram (2) showing an operation sequence in the embodiment of the present invention. FIG. 3 is a diagram (3) showing an operation sequence in the embodiment of the present invention. The figure explaining the operation | movement which detects an angle from the positional relationship of a camera and a display part of this invention. The figure explaining creation of 3D character images, such as a 3D avatar. The figure explaining the outline | summary at the time of using multiple cameras of this invention. The figure which shows the system configuration | structure of the 2nd Example of this invention. The figure explaining the operation | movement which detects a three-dimensional position using multiple cameras of this invention. The figure explaining the image processing in the case of displaying a person and another image simultaneously of this invention. The figure explaining the image processing in the case of displaying a person and a text simultaneously of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Display part, 3 ... Control part, 4 ... Memory part, 5 ... Input-output part, 6 ... Image recognition process part, 7 ... Direction analysis part, 8 ... Image conversion process part, 100 ... Image with camera Display device.

Claims (20)

A camera that shoots a subject and outputs image data of the subject;
A control unit that generates a character image of the subject based on the image data;
A display unit for displaying a character image of the subject;
In an image display device comprising:
The control unit detects a deviation between the direction of the subject and the direction of the subject from the image data, and generates a character image in which the direction and the deviation of the direction are corrected;
The image display device, wherein the display unit displays a character image corrected by the control unit. A camera that shoots a subject and outputs image data of the subject;
A control unit that generates a character image of the subject based on the image data;
A display unit for displaying a character image of the subject;
In an image display device comprising:
The control unit detects a deviation between the direction of the subject and the direction of the subject based on a positional relationship between the camera and the display unit, and corrects the direction and the deviation of the direction. Generate an image,
The image display device, wherein the display unit displays a character image corrected by the control unit.   3. The image display device according to claim 1, further comprising a memory unit that stores the character image.   The image display device according to claim 1, further comprising: a transmission unit that transmits the character image generated by the control unit.   3. The image display device according to claim 1, further comprising a receiving unit that receives image data corrected by another image display device. The image display device according to claim 1 or 2,
A transmission unit for transmitting the character image generated by the control unit;
A receiving unit for receiving image data corrected in another image display device,
The image display apparatus, wherein the display unit displays image data received by the receiving unit.   3. The image display device according to claim 1, wherein the character image is a three-dimensional character image.   3. The image display device according to claim 1, wherein when the control unit determines that the direction and the shift in the direction do not need to be corrected, the control unit generates the character image without performing the correction. A characteristic image display device.   3. The image display device according to claim 1, wherein when the subject is photographed, the display unit displays the subject and text.   3. The image display device according to claim 1, further comprising a plurality of cameras for photographing the subject. A camera that shoots a subject and outputs image data of the subject;
A memory unit for holding data necessary for generating a three-dimensional character image of the subject;
A control unit that generates a three-dimensional character image of the subject based on the image data of the subject and data held in the memory unit;
A display unit for displaying a character image of the subject;
An image display device comprising: a transmission / reception unit capable of transmitting and receiving data of the three-dimensional character image;
The control unit includes an image recognition processing unit, a direction analysis unit, and an image conversion processing unit,
The image recognition processing unit detects a direction in which the subject is facing from the image data,
The direction analysis unit detects a shift in the direction in which the subject is facing from the direction data detected by the image recognition processing unit,
The image display processing apparatus generates the three-dimensional character image in which the direction and the shift in the direction are corrected. A camera that shoots a subject and outputs image data of the subject;
A memory unit for holding data necessary for generating a three-dimensional character image of the subject;
A control unit that generates a three-dimensional character image of the subject based on the image data of the subject and data held in the memory unit;
A display unit for displaying a character image of the subject;
An image display device comprising: a transmission / reception unit capable of transmitting and receiving data of the three-dimensional character image;
The control unit includes an image recognition processing unit, a direction analysis unit, and an image conversion processing unit,
The image recognition processing unit detects a direction in which the subject is facing based on a positional relationship between the camera and the display unit;
The direction analysis unit detects a shift in the direction in which the subject is facing from the direction data detected by the image recognition processing unit,
The image display processing apparatus generates the three-dimensional character image in which the direction and the shift in the direction are corrected.   13. The image display device according to claim 2, wherein the display unit is directed to the direction of the subject and the direction of the subject based on an angle between a direction of a central axis of the camera and a front direction of the display unit. An image display device characterized by detecting a shift in a certain direction and generating a character image in which the direction and the shift in the direction are corrected. The first camera to shoot the photographer who is the subject,
A second camera for photographing the direction of travel of the photographer;
An image comprising: a display unit configured to display on the same screen an image of the photographer photographed by the first camera and an image of the photographer traveling direction photographed by the second camera. Display device. The first camera to shoot the photographer who is the subject,
A second camera for photographing the direction of travel of the photographer;
A display unit that displays an image of the contour of the photographer photographed by the first camera and an image of the traveling direction of the photographer photographed by the second camera on the same screen. An image display device. A camera that shoots a subject and outputs image data of the subject;
An image display device capable of displaying text on the display unit while photographing, and a display unit that displays an image of the subject based on the image data;
In the case where the image of the subject and the text are displayed on the same screen, the display unit displays the subject image and a background image other than the subject in a single color, respectively. . A camera that shoots a subject and outputs image data of the subject;
An image display device capable of displaying text on the display unit while photographing, and a display unit that displays an image of the subject based on the image data;
An image display device characterized in that the color change of the text is performed by any one of a photographing time, a photographing time limit and the number of characters of the text, or voice detection. A camera that shoots a subject and outputs image data of the subject;
An image display device capable of displaying text on the display unit while photographing, and a display unit that displays an image of the subject based on the image data;
An image display device characterized in that the text scrolling is performed by any one of photographing time, photographing time limit and the number of characters of text, or detection of voice. Shoot the subject and output the image data of the subject,
Generating a character image of the subject based on the image data;
In the image display method for displaying the character image of the subject,
Detecting a deviation between the direction of the subject and the direction of the subject from the image data;
Generate a character image in which the direction and the deviation of the direction are corrected,
An image display method comprising displaying the corrected character image. Shoot the subject and output the image data of the subject,
Generating a character image of the subject based on the image data;
In the image display method for displaying the character image of the subject,
Detecting a shift between the direction of the subject and the direction of the subject based on the angle between the direction of the central axis of the camera that captures the subject and the front direction of the display unit that displays the character image;
Generate a character image in which the direction and the deviation of the direction are corrected,
An image display method comprising displaying the corrected character image.

Images