JP2010034652A - Multi-azimuth camera mounted mobile terminal apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile terminal apparatus for achieving the use of a video telephone for capturing a self-portrait and the use of a digital camera for capturing a scenery image in such a state that any mobile section is absent by one system camera and control processing means in the mobile terminal apparatus on which a camera is mounted, and for enabling a mobile terminal apparatus user to use a photographic angle without considering it in a wide azimuth direction visual field. <P>SOLUTION: The apparatus mounted with an optical system consisting of a mirror unit and a camera unit includes: an image capturing means for capturing peripheral images in two or more azimuths; a means for extracting the rectangular visual field image of an arbitrary azimuth and an elevation angle range from the imaged peripheral image, and for performing image processing; a display means for displaying the rectangular visual field image; and a means for creating the rectangular visual field image from the peripheral image by switching the azimuth range for capturing the self-image of the mobile terminal apparatus user and the azimuth range for capturing the scenery image as the azimuth which is utterly different from this. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

      The present invention relates to a mobile terminal device equipped with a camera mechanism and control and image processing for easily capturing, displaying, and mutually transmitting and receiving a self-portrait and a surrounding landscape image of a mobile terminal device user.

A mobile phone with a camera is the most widespread form of a portable terminal device. In such a conventional camera-equipped mobile phone, as a means for photographing a self-portrait in a video phone device application and a portrait photographing application, a camera mounted on the same side as the display means and a landscape image in a digital camera application are photographed. As a means, two systems of cameras mounted on the opposite side of the display means are mounted according to the application. In addition, images taken with a camera-equipped mobile phone may vary depending on whether the camera mounting position is shifted from the center axis of the terminal device or the movement of the mobile terminal device user or the movement of the hand. A phenomenon has been recognized in which an object to be photographed, such as a landscape, is projected out of the center of the display means.
In Patent Document 1, in a videophone device application, a face position is detected from a self-portrait photographed by a camera using face discrimination means (skin color area detection), and the arrangement is controlled so that the face position becomes the center of the display means.
In patent document 2, in a videophone device application, an appropriate range (up / down / left / right range, appropriate distance range) of a self-portrait that is a reference taken by a camera is stored in advance, and based on this, By emphasizing and displaying in a direction away from the display means, the mobile terminal user who sees the display means wants to adjust the position.
JP 2005-191760 JP 2008-060734 A

  Conventional portable terminal devices of the background art require two systems of camera means and control processing means for video phone or portrait photography applications for taking self-portraits and digital camera applications for taking landscape images. . Further, the means for correcting the face position to be detected and displayed at the appropriate position of the display means is applied in a very narrow range based on the amount of deviation due to the camera mounting position and the appropriate sound position of the mobile terminal device user (such as a caller). Means.

  The present invention has been made in view of such circumstances, and in a mobile terminal device equipped with a camera, a videophone for taking a self-portrait for a self-portrait with a single camera and control processing means, or portrait photography use and scenery A mobile terminal device that realizes the use of a digital camera for taking images without moving parts, and can be used by a mobile terminal device user (such as a caller) without being particularly conscious of the shooting position and range due to a wide azimuth field of view. The purpose is to provide.

  The portable terminal device according to claim 1 includes an optical system including a mirror unit and a camera unit, and includes an imaging unit that captures peripheral images (moving images and still images) in two or more directions, and an arbitrary one based on the captured peripheral images. A portable terminal device comprising means for extracting and processing a rectangular field image of the azimuth and elevation angle range, and a display means for displaying the created rectangular field image, wherein the azimuth range for capturing a self-portrait of the user of the portable terminal device And a means for creating a rectangular field of view image from the surrounding image by switching the azimuth range for capturing a landscape image which is the opposite direction to the landscape image.

  The portable terminal device according to claim 2 includes an optical system including a mirror unit and a camera unit, and includes an imaging unit that captures peripheral images (moving images and still images) in two or more directions, and an arbitrary one based on the captured peripheral images. A portable terminal device having means for extracting and processing a rectangular visual field image in the azimuth and elevation angle range, and a display means for displaying the created rectangular visual field image, the rectangular visual field having a desired two azimuth range or more from the peripheral image Means is provided for simultaneously creating images.

  The mobile terminal device according to claim 3 includes means for detecting the orientation and position of a face of a person image included in the peripheral image, and tracking the movement of the orientation and position of the face of the person image. Means for detecting, estimating, and controlling the required azimuth and elevation range of the rectangular field image based on the azimuth and position information.

  The portable terminal device according to claim 4 includes a reference region extraction unit that extracts one or more reference regions included in the peripheral image, and a unit that tracks the movement of the reference region between the peripheral images at a predetermined time interval. It is characterized by comprising means for detecting and applying the posture and motion information of the mobile terminal device itself.

  The portable terminal device according to claim 5 extracts a rectangular field-of-view image in an arbitrary azimuth and elevation angle range from image storage means for accumulating captured peripheral images and peripheral images read from a desired time passage position of the image storage means. An image processing means and a display means for displaying the created rectangular field image are provided, and a means for simultaneously creating a rectangular field image having a desired one or more azimuth range from the peripheral image is provided.

  According to another aspect of the present invention, there is provided a mobile terminal device including a communication unit that transmits a captured peripheral image.

  According to the first aspect of the present invention, a desired azimuth range and elevation range can be obtained from peripheral images (moving images and still images) having two or more directions captured by a single imaging unit composed of a mirror unit and a camera unit. By cutting out the pixels and processing the image, it is possible to switch and create a self-portrait image or a rectangular field image of the landscape in the opposite direction, which is easier than a structure that conventionally requires two cameras. It can be realized at low cost. In addition, due to the wide visual field range of peripheral images of two or more directions, the subject does not deviate from the visual field in normal use, and there is no need for the mobile terminal device user to worry about the shooting position as in the conventional case. It is possible to concentrate on the operation.

  According to the second aspect of the present invention, two or more desired azimuth ranges can be obtained from peripheral images (moving images and still images) having two or more orientations captured by a single imaging means including a mirror unit and a camera unit. By cutting out the pixels in the elevation angle range and processing the image, it becomes possible to create two or more rectangular fields of view of the self-portrait and other orientations at the same time, compared to the conventional structure that requires multiple cameras. Can be realized easily and inexpensively. In addition, due to the wide visual field range of peripheral images of two or more directions, the subject does not deviate from the visual field in normal use, and there is no need for the mobile terminal device user to worry about the shooting position as in the conventional case. It is possible to concentrate on the operation.

  According to the third aspect of the present invention, a single image pickup means composed of a mirror part and a camera part simultaneously picks up peripheral images (moving images and still images) in two or more directions, and provides a wide field of view of multi-directional peripheral images. By detecting and tracking the face position and orientation in the range by face recognition processing, the user of the mobile terminal device can always change the position and orientation of the face (person) as well as the rectangular field image of the position and orientation of the face (person). It is possible to use a rectangular field-of-view image with a different orientation as a reference for display or communication.

  According to the fourth aspect of the present invention, peripheral images (moving images and still images) in two or more directions are simultaneously captured by one system of imaging means composed of a mirror unit and a camera unit, and the multi-directional directions are obtained from the peripheral images. By extracting the motion vector, it is possible to detect the relative movement and orientation of the mobile terminal device, and use this information for various applications such as switching the display screen by shaking the mobile terminal device. It becomes possible.

  According to the invention of claim 5, peripheral images (moving images and still images) of two or more directions simultaneously captured by one system of imaging means composed of a mirror unit and a camera unit are accumulated in the storage unit and accumulated. It is possible to read out the surrounding image at any time from a desired position (time or number of frames) in the storage unit and create a rectangular field image of a desired azimuth and elevation range, and a rectangle of the azimuth range displayed at the time of shooting Regardless of the visual field image, it is possible to obtain a rectangular visual field image having a desired orientation by going back in the accumulated time range at any time.

  According to the sixth aspect of the present invention, peripheral images (moving images and still images) in two or more directions that are simultaneously imaged by one system of imaging means composed of a mirror part and a camera part are transmitted to another person via communication means. It is possible to create a rectangular visual field image having a desired azimuth and elevation angle range from surrounding images received by others.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 illustrates the principle of a mobile terminal device according to a first embodiment of the present invention. FIG. 2 is a diagram illustrating the mirror unit of the mobile terminal device according to the first embodiment of the present invention. FIG. 3 is a diagram for explaining the flow of image acquisition of the mobile terminal device according to the first embodiment of the present invention. FIG. 4 is a diagram for explaining a display image selection method of the mobile terminal device according to the first embodiment of the present invention.

  In FIG. 1, the mobile terminal device 1 includes an imaging unit 2 including a mirror unit and a camera unit that simultaneously capture peripheral images (moving images or still images) in two or more directions, and two or more directions captured by the imaging unit 2. Image processing means for extracting a rectangular field image in an arbitrary azimuth and elevation range from the peripheral image of the image by image processing, and a display unit 3 for displaying the rectangular field image, and a rectangle having two or more desired azimuth ranges from the peripheral image A field-of-view image can be created at the same time. In the figure, the direction in which the mobile terminal device 1 and the mobile terminal device user face each other is defined as a front direction, and the opposite direction opposite thereto is defined as a back direction, and this direction is defined as an X direction. Moreover, the left-right direction of the portable terminal device 1 is defined as the Y direction, and the up-down direction is defined as the Z direction.

  The imaging means 2 has only two azimuths (front azimuth 5 and rear azimuth 7) in which the mirror is optically opposite in the surrounding imaging area 4 around the portable terminal device 1 due to the optical shape of the mirror. It is possible to take a picture from a partial azimuth range as a visual field to an omnidirectional range where the mirror unit optically covers the entire surrounding imaging region 4 as a visual field. In FIG. 2, the optical shape of the mirror portion will be described with reference to main examples. (A) The two-azimuth mirror portion (triangular prism) is an optical shape having a field of view of two directions obtained by bonding two plane mirrors of direction A and direction B. (B) The two-azimuth mirror portion (hyperbolic cross section) is an optical shape having a field of view with two azimuths of azimuth A and azimuth B having a mirror surface having a hyperbolic shape in the X-axis cross section. (C) The four-azimuth mirror portion (four-pyramid) is an optical shape having a field of view of four directions of an orientation A, an orientation B, an orientation C, and an orientation D each having four mirror planes in a quadrangular pyramid shape. (D) The omnidirectional mirror portion (hyperboloid) is an optical shape with the omnidirectional E as the field of view by the mirror shape forming the hyperboloid.

  The image pickup means 2 has a front direction 5 from the direction 6 where the mobile terminal device 1 and the mobile terminal device user face each other at least in the case of shooting a self-portrait such as videophone device use and portrait shooting due to the optical shape of the mirror part. When viewing landscape images such as digital camera applications, the visual field in the rear direction 7 from the direction 8 opposite to the mobile terminal device user direction of the mobile terminal device 1 is simultaneously captured as the same peripheral image. Can do. Each of the peripheral images including the front-view 5 view area and the rear-view 7 view area is displayed on the display unit 3 by subjecting one or both of the self-image 9 and the landscape image 10 to a rectangular view image. . In the image processing, the self-portrait 9 or the landscape image 10 is selected according to the usage state of the mobile terminal device 1 of the mobile terminal device user based on the peripheral images including the front-view 5 viewing area and the rear-view 7 viewing area. These rectangular visual field images may be switched and processed and displayed on the display unit 3. Further, since the peripheral image captured by the imaging unit 2 can be captured up to the omnidirectional visual field range of the peripheral imaging region 4 by the optical shape of the mirror portion, the image processing unit can detect the peripheral image to be captured. A rectangular field-of-view image having two or more desired azimuth ranges can be created at any time within the allowable azimuth range. The rectangular field-of-view image display on the display unit 3 is a screen (multi-screen) for a number of azimuths from a single screen, and other vertical and horizontal image processing is possible. In addition, when it is limited to the video phone device application, the rectangular field-of-view image display of the self-portrait 9 is the display unit 3 of the mobile terminal device 1 on the communication partner side.

  The mobile terminal device 1 can always obtain a self-portrait such as a videophone device and a landscape image such as a digital camera as a single image pickup means 2 and includes an omnidirectional view field range. The user can concentrate on communication and input operations without worrying about the camera angle of the mobile terminal device 1.

  FIG. 3 is a diagram for explaining the flow of image acquisition of the mobile terminal device according to the first embodiment of the present invention. (A) is a figure explaining the flow until it obtains a rectangular visual field image from the peripheral image (moving image or still image) which makes only two azimuth | directions (front azimuth | direction 5 and back azimuth | direction 7) a visual field range. (B) is a figure explaining the flow until it obtains a rectangular visual field image from the peripheral image (a moving image or a still image) which hits the peripheral imaging area 4 which has a visual field range as all directions.

  In (a), the image pickup means 2 is a camera that picks up the opposite azimuth constituted by the mirror section 11a (XZ plane view and XY plane view), and for example, picks up a mirror lens having a hyperbolic curve from below. Thus, it is possible to simultaneously capture images in the visual field ranges of the azimuth directions 11b and 11c opposite to a certain elevation angle range. The mirror unit 11a in the imaging unit 2 may have a two-sided mirror configuration that enables imaging in two directions, and may have any structure as long as a peripheral image with opposite directions can be obtained. In addition, the mirror unit 11a in the image pickup means 2 can be adapted to applications such as a triangular pyramid structure with a field of view in three directions, a quadrangular pyramid structure with a field of view in four directions, and a hyperboloid structure with a field of view in all directions. Selectable.

  In the peripheral image 12a, 13a of the opposite azimuth field imaged on the imaging surface by the imaging unit 2, the rectangular field images 14b, 15b in which the imaging area of the five front azimuth fields and the subject in the seven azimuth fields are shown are Each of the regions indicated by 14a and 15a is converted by image processing. In addition, the peripheral images 12a and 13a having the opposite azimuth field are converted into rectangular field images 12b and 13b by image processing, and the self-portrait 14b and the landscape image 15b, which are areas in which the subject is captured, are extracted. And 10 may be obtained. A hatched area 16 that is an area other than 12b and 13b is an orientation outside the field of view in the image pickup means 2 constituted by the mirror portion 11a, and indicates that no peripheral image exists.

  In (b), the imaging means 2 is composed of an optical mechanism portion 17a (XZ plane view and XY plane view), and is a camera that images the surrounding imaging area 4 with the omnidirectional field of view as the visual field range. By imaging the mirror lens from below, it is possible to obtain a peripheral image in a certain elevation angle range and a visual field range in all directions 17b. The mirror unit 17a in the imaging unit 2 may be a conical mirror that can capture the surrounding imaging region 4 having a field of view in all directions, and a peripheral image having the entire surrounding imaging region 4 as a field of view can be obtained. Any structure can be used.

  In the omnidirectional peripheral image 18a of the circular area shape imaged on the imaging surface by the imaging means 2, the front azimuth 5 shooting area and the rear azimuth 7 shooting area are imaged as fan-shaped areas indicated by 19a and 20a, respectively. Next, the image processing collectively converts the omnidirectional peripheral image 18a to the panoramic image 18b (omnidirectional rectangular field image), and extracts the target image 19b and the landscape image 20b of the target rectangular field image from the panoramic image 18b. The display images 9 and 10 can be created. Further, in order to reduce the load due to the amount of image processing, the rectangular field image 19b in the designated azimuth range from the fan-shaped image 19a in the five front azimuth areas and the sector image 20a in the rear azimuth seven photographing area are converted into the rectangular field image 20b in the designated azimuth range. Alternatively, the images 9 and 10 for display may be obtained by image processing individually.

  FIG. 4 is a diagram for explaining a display image selection method of the mobile terminal device according to the first embodiment of the present invention. First, in step S50, the mobile terminal device 1 captures a surrounding image with a omnidirectional field of view from a peripheral image (a counter azimuth peripheral image) having only two azimuth directions (front azimuth 5 and rear azimuth 7) as a visual field range. Input of any one of the peripheral images corresponding to the region 4 (omnidirectional peripheral images) is started. Next, in order to obtain a desired rectangular field-of-view image from these peripheral images (any one of the counter-azimuth peripheral image and the omnidirectional peripheral image) by image processing, the azimuth range and elevation angle range in step S51 are determined. When photographing a self-portrait such as a videophone device application, the azimuth range and the elevation angle range are set to be the designated azimuth range in the front azimuth 5 photographing region, and the process proceeds to step S53. When taking a landscape image such as for a digital camera, the azimuth range and the elevation angle range are set to be the designated azimuth range in the rear azimuth 7 shooting area, and the process proceeds to step S52. When the mobile terminal device user uses the operation input function of the mobile terminal device to arbitrarily specify the azimuth range and the elevation angle range in the possible azimuth range of the surrounding imaging region 4 and select to display an arbitrary azimuth image Proceeds to step S54. Note that step S54 is used when a rectangular field image of three or more directions is required, and by specifying a plurality of azimuth ranges, processing is performed so that the rectangular field images of each azimuth range are connected seamlessly. Things are also possible. Further, the user of the mobile terminal device can select to display the counter-polar azimuth peripheral images 12a and 13a or the omnidirectional peripheral image 18a captured by the imaging means 2 using the operation input function of the mobile terminal device. In this case, the process proceeds to step S55. In the case of steps S52, S53, and S54, in step S56, each surrounding image is subjected to image processing with a desired azimuth range and elevation angle range, and converted into a rectangular field image. In step S55, in step S57, the counter-azimuth peripheral images 12a and 13a or the omnidirectional peripheral image 18a are reduced to a rectangular visual field image by image processing or converted into a rectangular visual field image by cutting out a region of interest. . Further, step S57 can be processed so that the counter-azimuth peripheral images 12a and 13a or the omnidirectional peripheral image 18a are displayed in a split or scroll display. Finally, each rectangular field image is displayed on the display unit 3 in step S58.

  FIG. 5 is a diagram for explaining the display position adjustment by face recognition of the mobile terminal device according to the second embodiment of the present invention.

  FIG. 5A is a diagram for explaining face recognition in a peripheral image in a visual field range from two opposite directions to all directions. (B) is a figure explaining the face recognition in the panoramic image which image-processed the peripheral image of the visual field range from two opposite directions to all directions into a rectangular visual field image. Conventionally, when taking a self-portrait such as for a videophone device, the user of the mobile terminal device has always had to pay attention so that his / her face falls within the camera angle. In the mobile terminal device of the present invention, it is possible to always capture a peripheral image in a wide visual field range, and it is possible to sufficiently cover a range where the mobile terminal device moves due to camera shake or the like. In addition, when taking a self-portrait such as for a videophone device, the face position and orientation of the user of the mobile terminal device are detected from peripheral images having a wide field of view, and the image is based on the face position and orientation. By adjusting the azimuth range and the elevation angle range to be processed, it is possible to perform control so that the user of the mobile terminal device is always arranged in the rectangular visual field image. Further, the face recognition in the peripheral image can realize the same effect even when the user of the mobile terminal device shoots a landscape image when a person to be imaged exists in the landscape image.

  Further, when controlling the self-portrait of the user of the mobile terminal device or the person image (face image) in the landscape image to be arranged at the center of the rectangular field image, the face image is perpendicular to the vertical line of the rectangular field image. If there is a tilt component, the correction can be made in a direction in which the tilt component is eliminated.

  In addition, it detects the face position and orientation of the user of the mobile terminal device from the surrounding images, and uses the face position and orientation as a reference to display a rectangular field image corresponding to a landscape image in the azimuth range and elevation range opposite to the face position and orientation. It can be created by processing. Thereby, the mobile terminal device user changes the face position and orientation of the mobile terminal device so that the user can specify the position and orientation of the landscape image that he / she sees over the mobile terminal device. A field-of-view image can be obtained. Note that the user of the mobile terminal device can arbitrarily set not only the opposite azimuth based on the face position and azimuth, but also the position and azimuth that differ from each other by the relative angle when acquiring the rectangular field-of-view image. Further, by detecting up to the line-of-sight position and orientation of the user of the mobile terminal device in face recognition, it is possible to obtain a rectangular field-of-view image having the position and direction indicated by the line of sight of the user.

  The face recognition processing of the present invention is based on the optical shape of the mirror part of the image pickup means 2, and the mirror part of the surrounding imaging region 4 around the mobile terminal device 1 is optically opposite to the two opposite directions (front azimuth 5 And a peripheral image of a partial azimuth range with only the rear azimuth 7) as a field of view, and a peripheral image of an omnidirectional range with the mirror portion optically viewing the entire surrounding imaging region 4 as a field of view. In the example of (a), the peripheral image 12a is a peripheral image (omnidirectional peripheral image) captured by the imaging unit 2 on the imaging surface in the omnidirectional visual field range. Here, 21a and 22a are self-portraits of the mobile terminal device user taken in the omnidirectional peripheral image 12a, and indicate that the mobile terminal device has moved from 21a to 22a over time. Here, the frames 21b and 22b indicate the positions where the face positions are recognized in the respective self-portraits, and it is understood that the movement amount is the declination component 23a indicated by the polar coordinate system and the change 23b in the radial position. The movement amount 23a is an azimuth angle component moved by the user of the mobile terminal device, and the movement amount 23b is an elevation angle component moved by the user of the mobile terminal device, and the azimuth range and the elevation angle range for image processing based on the face position are determined. By adjusting, it is possible to control so that the user of the mobile terminal device is always arranged in the rectangular field image. In addition, even when a peripheral image of an omnidirectional visual field, such as a peripheral image of a two-directional visual field range, is not captured, the same is realized in the permissible visual field azimuth range.

  In the example of (a), the face position and orientation of the user of the mobile terminal device are detected from the surrounding images, and the landscape of the azimuth range and the elevation angle range opposite to the face position and azimuth based on the face position and orientation. When a rectangular field image corresponding to an image is created by image processing, the orientation of the landscape image changes as shown in 23c along with the change of the face position and orientation. In (b), the same processing can be performed because the horizontal axis corresponds to the azimuth angle component.

  In (b), a panoramic image (omnidirectional rectangular field image) 12b obtained by converting a peripheral image in the omnidirectional visual field range into a rectangular field image is an image created by image processing from the omnidirectional peripheral image 12a. Reference numerals 24a and 25a denote self-portraits of the user of the mobile terminal device captured in the omnidirectional rectangular visual field image 12b, and indicate that the user has moved from 24a to 25a over time. Here, the frames 24b and 25b indicate positions where the face positions are recognized in the respective self-portraits, and it is understood that the movement amounts are the horizontal axis component 26a and the vertical axis component 26b. The movement amount 26a is an azimuth movement component moved by the mobile terminal device user, and the movement amount 23b is a movement component in the depth direction moved by the mobile terminal device user. By adjusting the horizontal axis position (azimuth position) and vertical axis position (depth position) of the image extracted based on this face position, the image is always arranged in the rectangular field image displayed by the mobile terminal device user. It becomes possible to control. First, image processing from the omnidirectional peripheral image 12a to the omnidirectional rectangular field image 12b is required in the range from the prospective region to the entire region, and there is an overhead compared to (a), but in the orthogonal coordinate system. There is also an advantage that the processing can be performed.

  FIG. 6 is a diagram for explaining the flow of stored image display in the portable terminal device according to the third embodiment of the present invention.

  In the portable terminal device according to the present invention, the peripheral image area 4 having the omnidirectional field of view as the visual field range corresponds to the peripheral image area 4 having the omnidirectional field of view as the visual field range. A storage unit for storing a peripheral image determined by the optical shape of the mirror unit in the imaging means 2 in the range of the peripheral image (omnidirectional peripheral image) is provided, and anytime from the peripheral image read from a desired position of the storage unit It is possible to switch between the azimuth range for photographing a self-portrait of a mobile terminal device user and the azimuth range for photographing a landscape image, and to perform a desired rectangular field-of-view image on the same peripheral image and display it on the display unit. Become. The image format stored in the storage unit may be any of a non-compressed format (YUV format, RGB format, etc.) and a compressed format (MPEG2 format, H.264 format, JPEG format, etc.).

  The user of the mobile terminal device can set whether to store the peripheral image captured by the imaging unit 2 in the storage unit by operating the mobile terminal device in advance. When there are images (moving images and still images) stored in the storage means, the mobile terminal device user can display the images to be displayed on the display unit 3 retrospectively in time series. In step S60, the user of the mobile terminal device selects whether to display the latest image at the present time or to display it retroactively. When displaying the latest image, the latest image position in the storage means is acquired, read and displayed in step S63. When displaying retroactively in the past, the time position starting in step S61 is designated in step S62, or step S65 in which the next image is displayed in time series is selected. In step S62, the user of the mobile terminal device can use the operation input means (operation keys, touch panel, etc.) to specify the image position within the range from the latest image to the oldest image depending on the storage capacity of the storage means. Next, in step S64, the image reading position at the time position specified by the mobile terminal device user is set. Next, in step S66, a peripheral image (peripheral azimuth peripheral image to any omnidirectional peripheral image) is read from the storage means from the designated position, and the portable terminal device user's own image is photographed in steps S67 and S68. By switching between the azimuth and elevation angle range to be taken and the azimuth and elevation angle range for taking a landscape image, a desired rectangular field-of-view image can be created by image processing from the read peripheral image, and the display step S69 can be performed. In addition, the user of the mobile terminal device may simultaneously create a rectangular visual field image having two or more desired azimuth and elevation angle ranges by image processing.

  FIG. 7 is a diagram illustrating the posture information of the mobile terminal device according to the fourth embodiment of the present invention. FIG. 8 is a diagram for explaining the reference area of the mobile terminal device according to the fourth embodiment of the present invention. FIG. 9 is a diagram for explaining posture detection of the mobile terminal device according to the fourth embodiment of the present invention.

  In FIG. 7, the mobile terminal device according to the present invention has a peripheral area in which all directions are from a peripheral image (opposite direction peripheral images) having only two opposite directions (front direction 5 and rear direction 7) as a visual field range. From the peripheral image determined by the optical shape of the mirror portion in the imaging means 2 in the range of the peripheral image (omnidirectional peripheral image) corresponding to the imaging region 4, the parallel movement component in the orthogonal coordinates indicated by the XYZ axes in the drawing of the mobile terminal device itself It is possible to detect the rotation component around the Z axis and the tilt component of the plane including the X and Y axes.

  In FIG. 8, (a) is a reference region for detecting posture information of the mobile terminal device itself in a peripheral image 40 (omnidirectional peripheral image) captured on the imaging surface by the imaging means 2 in the omnidirectional visual field range. FIG. (B) is for detecting movement and posture information corresponding to (a) in an image obtained by converting the peripheral image 40 (omnidirectional peripheral image) into a rectangular visual field image 41 (omnidirectional rectangular visual field image) by image processing. It is a figure which shows the position and shape of a reference | standard area | region. Each of the reference regions in (a) and (b) corresponds to a region that covers the same azimuth angle range and elevation angle range, although the shapes are different. Therefore, any form of the omnidirectional peripheral image 40 (a) and the omnidirectional rectangular field image 41 (b) may be used as the reference area in the movement and posture detection. The reference areas 42a to 49a are assigned to specific radial ranges and declination ranges of the polar coordinate system in the omnidirectional peripheral image 40, and almost all azimuth movement and posture components can be detected by one or more of these reference areas. To be secured. Similarly, the reference regions 42b to 49b in the omnidirectional rectangular field image 41 can detect movement and posture components in the same direction corresponding to 42a to 49a. These movement and posture components can be detected by extracting motion vectors in each region between time-series images at fixed time intervals in the omnidirectional peripheral image 40 or the omnidirectional rectangular visual field image 41. In addition, areas that have been identified by human face recognition by face recognition should be excluded from areas that have behaviors different from the movement and posture of the mobile terminal device, such as excluding them from the reference area in advance. It is also possible to make it.

  In FIG. 9, (a) is a figure explaining the attitude | position of the imaging means 2 (mirror part). (B) is a figure explaining the movement and attitude | position information of a portable terminal device to detect. The figure of (a) is the rotation component 71 centering on the Z-axis in the three-dimensional space (XYZ axis) of the mirror part 70 which comprises the imaging means 2, the parallel movement component 72 on the X-axis, and the inclination component of an XY-axis plane. 73. Arrows described in the omnidirectional peripheral image 71a in (b) indicate movement and posture components detected in the reference regions 42a, 44a, 46a, and 48a when the mirror unit 70 has a rotation component 71 centered on the Z axis. Is a vector. The arrows described in the omnidirectional rectangular field image 71b are the movement and orientation component vectors detected in the reference regions 42b, 44b, 46b, and 48b when the mirror unit 70 has the rotation component 71 centered on the Z axis. . The arrows described in the omnidirectional peripheral image 72a are movement and posture components detected in the reference regions 42a, 44a, 46a, and 48a when the mirror unit 70 has the parallel movement component 72 on the X axis. The arrows described in the omnidirectional rectangular field image 72b are movement and posture components detected in the reference regions 42b, 44b, 46b, and 48b when the mirror unit 70 has the parallel movement component 72 on the X axis. The arrows described in the omnidirectional peripheral image 73a are movement and posture components detected in the reference areas 42a, 44a, 46a, and 48a when the mirror unit 70 has the XY axis plane tilt component 73. The arrows described in the omnidirectional rectangular field image 73b are the movement and posture components detected in the reference regions 42b, 44b, 46b, and 48b when the mirror unit 70 has the tilt component 73 of the XY axis plane. The XYZ axis movement and orientation information of the mobile terminal device itself in the mirror unit 70 is an integrated analysis of vector components in each direction as shown in the symbolic movement and orientation component vectors in these reference areas. Can be easily estimated.

  The movement information and posture information of the mobile terminal device itself are estimated from these movement and posture components detected in the omnidirectional peripheral image or the omnidirectional rectangular visual field image, and the mobile terminal device is shaken using the movement and posture information. Can be used for various applications such as switching the display screen. Further, by dividing the movement information of the mobile terminal device itself to be extracted by the time series image interval (time) for measuring, it is possible to derive the relative speed and the acceleration as a derivative thereof.

  FIG. 10 is a diagram for explaining the communication function of the mobile terminal device according to the fifth embodiment of the present invention.

  In the figure, the mobile terminal device 1a is connected to the other mobile terminal device 1b via the communication means 80 in the peripheral image (counterpolar azimuth peripheral image) having only two opposite azimuth directions (front azimuth 5 and rear azimuth 7). ), It is possible to transmit a peripheral image determined by the optical shape of the mirror part in the imaging means 2a in the range of the peripheral image (omnidirectional peripheral image) including the entire peripheral imaging region 4 having the omnidirectional visual field range. It is. In addition, the mobile terminal device 1a receives from the other mobile terminal device 1b via the communication unit 80 a peripheral image (counterpolar azimuth peripheral image) having only two opposite azimuth directions (front azimuth 5 and rear azimuth 7). Thus, it is possible to receive the peripheral image determined by the optical shape of the mirror part in the imaging means 2b in the range of the peripheral image (omnidirectional peripheral image) including the entire peripheral imaging region 4 having the omnidirectional visual field range. is there. The mobile terminal device 1a can perform image processing on the received peripheral image 82a according to a desired azimuth range and elevation angle range, convert the image into rectangular visual field images (85a and 85b), and display them on the display means. In addition, by transmitting the peripheral image via the communication unit 80, the mobile terminal device on the receiving side can always display a rectangular visual field image in a desired azimuth range and elevation angle range in the azimuth visual field range available for the peripheral image. It can be created by image processing. For example, the mobile terminal device 1a can convert a desired region, such as 83a which is a self-portrait of 81b from the received peripheral image 82a or 84a which is a landscape image, into a rectangular field image (85a and 85b) by image processing. In addition to the mobile terminal device, the communication partner device 1b may be a general-purpose electronic computer such as a personal computer, a video-related device having a communication function, or a large-sized electronic computer such as a server.

  According to the mobile terminal device of the embodiment of the present invention, at least one self-portrait photographed for videophone device use and two or more opposite images such as landscape images photographed for digital camera use are once captured by one camera. It is possible to shoot at the same time. In addition, it is possible to shoot a peripheral image with a wide azimuth field of view, and the mobile terminal device user's self-portrait will not be inadvertently removed from the frame for videophone applications etc. It is possible to control the terminal device user's face to be arranged in the display unit image.

The figure explaining the principle of the portable terminal device which concerns on 1st Example of this invention The figure explaining the mirror part of the portable terminal device which concerns on 1st Example of this invention. The figure explaining the flow of the image acquisition of the portable terminal device which concerns on 1st Example of this invention. The figure explaining the selection method of the display image of the portable terminal device which concerns on 1st Example of this invention. The figure explaining the display position adjustment by the face recognition of the portable terminal device which concerns on 2nd Example of this invention. The figure explaining the flow of the memory | storage image display in the portable terminal device which concerns on 3rd Example of this invention. The figure explaining the attitude | position information of the portable terminal device which concerns on 4th Example of this invention The figure explaining the reference | standard area | region of the portable terminal device which concerns on 4th Example of this invention The figure explaining the attitude | position detection of the portable terminal device which concerns on 4th Example of this invention The figure explaining the communication function of the portable terminal device which concerns on 5th Example of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Mobile terminal device 2 Camera part 3 Display part 4 Peripheral imaging area 5 Front direction imaging area 6 Direction which a mobile terminal device user faces 7 Back direction imaging area 8 Direction opposite to a mobile terminal device user direction 9 Self-portrait 10 Landscape image

Claims (6)

An imaging unit that includes an optical system including a mirror unit and a camera unit, and that captures peripheral images in two or more directions;
Means for extracting a rectangular visual field image in an arbitrary azimuth and elevation angle range from the captured peripheral image and image processing;
A portable terminal device comprising display means for displaying the rectangular field-of-view image,
Switch between the azimuth range for taking a self-portrait of a mobile terminal device user and the azimuth range for taking a landscape image, which is the opposite direction to this.
A portable terminal device comprising means for creating the rectangular field-of-view image from the peripheral image. An imaging unit that includes an optical system including a mirror unit and a camera unit, and that captures peripheral images in two or more directions;
Means for extracting a rectangular visual field image in an arbitrary azimuth and elevation angle range from the captured peripheral image and image processing;
A portable terminal device comprising display means for displaying the rectangular field-of-view image,
A portable terminal device comprising means for simultaneously creating the rectangular field-of-view image having two or more desired azimuth ranges from the peripheral image. The portable terminal device according to claim 1, wherein:
Detecting the orientation and position of the face of the human figure included in the peripheral image,
Means for tracking the movement of the orientation and position of the face of the person image,
A portable terminal device comprising means for detecting, estimating, and controlling a required azimuth and elevation range of a rectangular field-of-view image based on a specific person's face orientation and position information. The portable terminal device according to claim 1, wherein:
Reference area extraction means for extracting one or more reference areas included in the peripheral image;
By comprising means for tracking the movement of the reference area between the peripheral images at a certain time interval,
A portable terminal device comprising means for detecting and applying the posture and motion information of the portable terminal device itself. The portable terminal device according to claim 1, wherein:
Image storage means for storing the captured peripheral image;
Means for extracting and processing a rectangular visual field image in an arbitrary azimuth and elevation angle range from the peripheral image read from a desired time passage position of the image storage means;
Comprising display means for displaying the rectangular field-of-view image,
A portable terminal device comprising means for simultaneously creating the rectangular field image having a desired one azimuth range or more from the peripheral image. The portable terminal device according to claim 1, wherein
A portable terminal device comprising communication means for transmitting the captured peripheral image.