JP2020085914A - Information terminal with camera and program - Google Patents

Abstract

To provide an information terminal with a camera that performs a zoom operation with an icon gazed at after a prescribed action of an eye ball and head.SOLUTION: An information terminal with a camera has: a field-of-view camera (11) that is mounted to a face side of a user; a posture change detection unit (13) that detects a posture change of the field-of-view camera (11); an eyeball camera (14) that photographs an eyeball of the user; a line-of sight detection unit (15) that detects a line of sight from an image of the eyeball camera (14); and a display part (16) that can be viewed by the user. When a direction of the posture change of the field-of-view camera (11) the posture change detection unit (13) and line-of sight detection unit (15) detect respectively, and a direction of a line-of-sight change are a cancelling-out direction (STP4), a gaze icon (IC) is displayed around a center of the display part (16) (STP5), and a zoom rate of the field-of-view camera (11) is controlled so as to be large or small while a viewpoint of the user stays at the gaze icon (IC) (STP11).SELECTED DRAWING: Figure 8

Description

The present invention relates to a camera-equipped information terminal mounted on a head and a program.

2. Description of the Related Art There is known a user interface that controls a device based on the direction of the head including the face and the direction of the line of sight. For example, Patent Document 1 describes a head mounted display (1) including a display (11) on which an icon (described as a pointer in the document) is displayed. This is a head-mounted device that includes a camera (12) that detects the direction of the line of sight of the user and an acceleration sensor (13) that detects the movement of the head of the user. This device is a pointing system that allows the user to swing or move his/her head up/down or left/right while keeping his/her eyes on the icon, so that the user can click or drag the mouse to operate the computer. Since the relative position of the user's eyeball and the display (1) does not change, the eyeball does not rotate when the head is shaken unless the position of the icon in the display (1) changes. That is, there is no change in the line of sight. When the user removes the viewpoint from the icon, even if the user shakes his/her head, it is not regarded as a click motion, and an erroneous operation is prevented with respect to the motion of unconsciously shaking the head.

Further, in Patent Document 2, although not a head-mounted device, a user retains a viewpoint (described as a line of sight in the document) on an icon (30) of a display unit (20) of a display placed on a desk. There is described a gesture recognition device that judges that the icon (30) is selected by performing an operation of tilting the head back and forth while moving the head (described as a nod gesture). This is to detect the line of sight of the user and the direction of the face in front of the head with the camera 10 provided on the upper portion of the display unit (20). In this device, the positional relationship between the user's head and the display unit (20) is not relatively fixed. Therefore, if the user makes a nod gesture with the viewpoint retained on the icon (30), the user's eyeballs rotate, and the direction of the head and the direction of the line of sight change to directions that do not match. Become. As described above, Patent Document 3 also describes a method of indicating an intention by a disagreement operation between the direction of the line of sight and the direction of the face.

Further, Patent Document 4 describes an imaging device that controls a zoom operation of a camera by a user gazing a predetermined mark on a finder screen (102). This is because the indexes (201, 202) indicating the wide side and the tele side of the zoom operation are displayed in the corners of the finder screen (102), and when the user looks at the display portion of the desired zoom ratio index for a predetermined time. The zoom operation of the imaging device is performed. A mark called a zoom ratio index is provided in a corner of the finder screen (102) where the user does not normally look, and this mark is used for the user interface to prevent a malfunction that is not intended by the user.

In addition, Patent Document 5 describes an image forming apparatus including a step (103) of enlarging the display of a portion of the touch-panel operation display unit (106) where the user's line of sight is facing. This is configured such that the enlarged display process is not performed when the user's finger is detected near the display unit (106). In this way, it is possible to prevent the operability from being undesirably enlarged and displayed when the user is performing a finger touch operation.

Further, Patent Document 6 describes a navigation device that enlarges and displays a screen by gazing at a plurality of predetermined points. In this way, the screen is enlarged and displayed only when the user looks at a plurality of points, thereby preventing the enlarged display against the user's intention. In this way, technology for operating the device based on the movement of the line of sight, face, and head has been disclosed. These are accompanied by a structure that does not operate contrary to the user's intention.

However, when these configurations are applied to the zoom operation of the head-mounted camera, the following problems occur. For example, if the configuration of Patent Document 1 is applied, it is necessary to shake the head up and down or left and right with some acceleration while gazing at the viewfinder of the camera or the icon displayed on the screen corresponding thereto. When the head is shaken, the camera attached to the head also shakes, which causes a problem that the object to be photographed cannot be captured properly.

Further, with the configurations described in Patent Document 2 and Patent Document 3, when the user's viewpoint is placed on the subject to be photographed, the orientation of the head is out of the subject to be photographed. Since the head-mounted camera matches the orientation of the head, the camera is also removed from the shooting target, and there is a problem that it is difficult for the user to perform a zoom operation while holding the target on the screen.

By the way, it is preferable that the center of the viewfinder of the camera or the screen corresponding thereto coincides with the center of the camera. This is because the user can easily understand the center and range of the image captured by the camera. If the configuration described in Patent Document 4 is applied to the camera based on this, it is unnatural that the zoom icon at the corner of the screen must be watched for the zoom operation while the shooting target is captured in the center of the screen. There is a problem of being forced to move. If the zoom icon is displayed near the center of the screen in order to avoid this, the user often looks at the vicinity of the center of the screen with high frequency, which causes a problem that the malfunction of the zoom operation tends to occur against the user's intention. ..

Further, if the configuration of Patent Document 5 is applied to a zoom operation of a head-mounted camera, in addition to the movements of the eyeball and the head, the movements of the hands such as fingers are required, and the freeness of both hands is ensured. In the first place, there is a problem that the original merit cannot be realized. Further, if the configuration of Patent Document 6 is applied to the same zoom operation, it is necessary to sequentially gaze at a plurality of icons on the screen, which causes a problem that the procedure is complicated and takes time.

JP, 2006-243784, A JP, 2000-163196, A JP, 2007-94619, A JP-A-8-46846 JP, 2018-18437, A JP, 2018-77105, A

Therefore, an object of the present invention is to provide a camera-equipped information terminal and a program that are capable of zoom operation while gazing at a subject to be photographed, which is an operation by eye movements and head movements while preventing malfunction. is there.

As an embodiment for solving the above-mentioned problems, a view camera (11) mounted on the user's face side, a posture change detection unit (13) for detecting a posture change of the view camera (11), and an eyeball of the user. An eyeball camera (14) for photographing the camera, a line-of-sight detection unit (15) for detecting a line of sight from an image of the eyeball camera (14), and a display unit (16) visible to the user, and a camera-equipped information terminal. can do. Then, when the orientation change direction of the view camera (11) detected by the posture change detection section (13) and the sight line detection section (15) respectively cancel out the direction of the sight line change, the display section A gaze icon (IC) is displayed in the vicinity of the center of (16), and the zoom ratio of the view camera (11) is controlled to be large or small while the user's viewpoint is retained on the gaze icon (IC). ..

Embodiments of the present invention can provide a camera-equipped information terminal and a program capable of performing zoom operations while gazing at an object to be photographed, which is an operation based on eye movements and head movements while preventing malfunction. ..

FIG. 1 is a diagram of an information terminal with a camera according to the first embodiment of the present invention. FIG. 2 is a diagram for explaining a photographing system and a display system of the information terminal with camera according to the first embodiment of the present invention. FIG. 3 is a diagram showing an equivalent view camera and an equivalent display unit of the information terminal with camera according to the first embodiment of the present invention. FIG. 4 is a diagram for explaining the relationship between the operation of the user and the display on the display unit. FIG. 5 is a diagram illustrating a gaze icon. FIG. 6 is a diagram showing the vicinity of the center of the display unit. FIG. 7 is a block diagram showing the configuration of the camera-equipped information terminal according to the first embodiment of the present invention. FIG. 8 is a process flow chart of the information terminal with camera according to the first embodiment of the present invention. FIG. 9 is a diagram for explaining an operation in which the line of sight of the user changes downward. FIG. 10 is a diagram showing an operation in which the zoom ratio is reduced. FIG. 11 is a diagram showing a processing flow of the camera-equipped information terminal according to the second embodiment of the present invention. FIG. 12 is a diagram illustrating a method of detecting a line of sight from an eyeball camera image. FIG. 13 is a diagram for explaining the operation of the posture change detection unit according to the first embodiment and the second embodiment. FIG. 14 is a diagram showing the configuration of the information system according to the third embodiment of the present invention. FIG. 15 is a diagram showing the relationship between the neutral viewpoint of the user and the center of the display unit and the center of the view camera (fourth embodiment).

[First embodiment Information terminal with camera]
With reference to FIG. 1, a camera-equipped information terminal 1 according to a first embodiment of the present invention will be described. FIG. 1A is a diagram showing a state in which the user wears the camera-equipped information terminal 1. (A2) shows the right eye side of the user. FIGS. 1B1 and 1B2 are views showing a part of the information terminal 1 with the camera in a transparent manner. As shown in FIG. 1, each component of the camera-equipped information terminal 1 is mounted on the user's head by a frame 1F. The right eye side is covered with a display unit cover 16A that covers the front of the right ear. Inside the display cover 16A, a visual field camera 11 that captures the foreground of the user, an eyeball camera 14 that captures the right eye of the user, and an image captured and acquired by the visual field camera 11 are visible to the user. A display unit 16 for displaying is provided. Further, in FIG. 1(B1), the display cover 16A is transparent. Inside this, a prism and a mirror necessary for display and photographing are included, but the details will be described with reference to FIG. The view camera 11 shown in FIG. 1(A2) is also mounted in a state in which it cannot be visually recognized.

A control box 2A is also mounted on the frame 1F. An arithmetic unit 21, a communication unit 22, and a memory 23, which will be described later in detail, are housed therein. The computing unit 21 performs a posture change detection process of detecting a posture change of the view camera 11 and a line-of-sight detection process of detecting a user's line of sight from an image acquired by the eyeball camera 14.

[Shooting system and display system]
The photographing system and the display system of the camera-equipped information terminal 1 will be described with reference to FIG. The optical path LP1 is guided to the view camera 11 by the imaging prism 16C and a moving image is captured. The photographing prism 16C is a substantially triangular prism, and is shown as a triangle in FIG. The second side of the three sides is composed of a mirror that reflects light for all incident angles. Therefore, the optical path LP1 of the light transmitting the foreground of the user does not directly enter the right eye R-Eye. The foreground of the user is captured by the view camera 11 and displayed on the display unit 16 in almost real time. The user can visually recognize his/her foreground through the display unit 16. The optical path LP2 of the light emitted from the display unit 16 is guided to the right eye R-Eye by the inner wall of the display unit cover 16A, the mirror 16B, and the viewing prism 16D. Further, among the surfaces of the prism 16D for visual recognition, the surface facing the prism 16C for photographing is composed of a mirror that reflects light for all incident angles. In this way, the captured image of the view camera 11 mounted on the head of the user is visually recognized by the user via the display unit 16 mounted on the head.

[Equivalent field of view camera and equivalent display section]
In the camera-equipped information terminal 1 according to the first embodiment described with reference to FIGS. 1 and 2, the view camera 11 and the display unit 16 are incorporated in the display unit cover 16A and are not visible. On the other hand, in the camera-equipped information terminal 1 shown in FIGS. 3A and 3B, the field-of-view camera 11 and the display unit 16 are shown outside. However, it will be explained that what has been described with reference to FIGS. 1 and 2 is equivalent to that shown in FIG. 3 in the following points.

First, the view camera 11 of the information terminal with camera 1 according to FIGS. 1 and 2 can acquire the foreground of the user's head. Then, the posture of the view camera 11 also changes as the posture of the user's head changes. From this point of view, the field-of-view camera 11 shown in FIGS. 3A and 3B is equivalent to the field-of-view camera 11 of FIGS. Further, the display unit 16 of the camera-equipped information terminal 1 according to FIGS. 1 and 2 displays the acquired image of the view camera 11. Even if the posture of the user's head changes, the relative positional relationship between the display unit 16 and the user's right eye R-Eye remains unchanged. In this respect, the display 16 shown in FIGS. 3A and 3B is equivalent to the display 16 according to FIGS. 1 and 2. Therefore, hereinafter, description will be made based on the view camera 11 and the display unit 16 shown in FIG.

By the way, in FIG. 3, the display unit 16 appears to block the front of the view camera 11. However, in the following description, it is assumed that the display unit 16 is out of the visual field of the view camera 11. This is because the camera-equipped information terminal 1 shown in FIGS. 1 and 2 avoids interference such as blocking the other from each other by the prism and the mirror. That is, in FIGS. 1 to 3, the field-of-view camera 11 is attached to the face side of the user so that the nose of the user faces, and is capable of photographing the foreground of the user. Further, the display unit 16 is mounted on the head so that the user can visually recognize it, and displays the acquired image of the view camera 16. Further, as a premise, it is assumed that the center of the image acquired by the view camera 11 is adjusted to be displayed at the center of the display unit 16. Then, when the view camera 11 performs a zoom operation, the center of the acquired image is used as a base point for enlargement or reduction.

[User action]
Next, with reference to FIG. 4, an important user operation in the embodiment will be described. FIG. 4 is a diagram for explaining a change in the image acquired by the view camera 11 and a change in the image displayed on the display unit 16 according to the operation of the user. 4(A1), (B1) and (C1) are diagrams showing the motion of the user. On the other hand, FIGS. 4(A2), (B2), and (C2) are images displayed on the display unit 16 and are also images captured by the view camera 11.

FIG. 4A1 shows a state in which the user is gazing at a small white square displayed in the center of the display unit 16 with his line of sight. That is, the line of sight VD of the user is toward the center of the display unit 16. Further, as described above, since the center of the image acquired by the view camera 11 has been adjusted to be displayed in the center of the display unit 16, the direction CD of the view camera 11 is the same as the direction of the line of sight VD of the user. ing. Here, for convenience of description, a point where the user's line of sight VD intersects the surface of the display unit 16 is referred to as a viewpoint VP.

FIG. 4(A2) is the image displayed on the display unit 16 at this time. The user is gazing at the small white square in the center of the display unit 16. A line segment that is horizontal to the screen of the display unit 16 and passes through the center is displayed as a camera line CL. Further, a line segment that passes through the viewpoint VP and is horizontal to the screen of the display unit 16 is displayed as a viewpoint line VL. In this case, the camera line CL and the viewpoint line VL match.

[Offset operation]
FIG. 4B1 shows a state in which the user leaves his/her viewpoint on a small square, which is the gazing point, and the posture of the head is slightly downward. This is also the state in which the user has pulled his chin a little. The field-of-view camera 11 mounted on the user's head and the display unit 16 are tilted at the same angle as the head. As a result, as shown in FIG. 4B2, the image displayed on the display unit 16 moves slightly upward on the screen. At this time, a difference occurs between the direction CD of the view camera 11 and the direction of the line of sight VD of the user. A difference also occurs between the camera line CL and the viewpoint line VL, which are represented as two separated line segments. As in the example shown here, when the posture of the head is changed while leaving the viewpoint VP in a specific portion of the foreground, the line of sight changes in a direction canceling the change in the posture of the head. Here, such an operation that causes a change in the line of sight in the direction opposite to the change in the posture of the head is also referred to as a canceling operation.

[Gaze icon]
4(C1) and FIG. 4(B1) have exactly the same posture. There is no change in the behavior of the user. Therefore, the image displayed in FIG. 4(C2) matches the image displayed in FIG. 4(B2). However, in FIG. 4C2, the gaze icon IC is displayed at the center of the display unit 16.

FIG. 5 is a diagram illustrating the gaze icon IC displayed on the display unit 16. FIG. 5(A1) is a user who has the same head posture as FIG. 4(C1). However, the viewpoint VP remains in the gazing icon IC. As a result, the orientation CD of the view camera 11 and the orientation of the user's line of sight VD match. On the screen of the display unit 16 shown in FIG. 5(A2), the camera line CL and the viewpoint line VL coincide. However, the viewpoint line VL is displaced downward from the small white square where the viewpoint VP was initially retained (see FIG. 4(A2)).

Here, while keeping the viewpoint VP in the gaze icon IC, it is easy to match the viewpoint VP with the small white square that was initially retained. The jaw may be slightly protruded while maintaining the retention of the viewpoint on the gaze icon IC. The state immediately after the execution of this operation is shown in FIG. 5(B1). As a result of this operation, as shown in FIG. 5B2, the image on the display unit 16 moves slightly downward, and the camera line CL, the viewpoint line VL, and the gaze icon IC overlap with the small white square. The posture of the user at this time coincides with that in FIG. 4(A1). The sequence of operations from FIG. 4(A1) to (B1), (C1), and further through FIG. 5(A1) to (B1) is not usually an operation unconsciously performed by the user. This is an action that is not performed unless it is conscious. Although this is an unnatural operation, the operation can be easily performed by displaying the gaze icon IC on the display unit 16 from FIGS. 4C2 to 5A2 and 5B2. Then, FIGS. 5C and 5D show that the zoom ratio is gradually increased and displayed with the center of the display unit 16, that is, the gaze icon IC as the center.

In principle, the gaze icon IC is displayed in the center of the display unit 16. However, the present invention is not limited to this, and the display unit 16 may have a spread from the center. For example, as shown in FIG. 6(A), it is a region that extends from the center of the display unit 16 to some extent. This is called a central area 161 and the gaze icon IC is displayed in this area. The central region 161 is shown in a larger size for clarity. As shown in FIG. 6(B), the line of sight at which the user gazes at DS0, which is the center of the display unit 16, is defined as VDP, and the line of sight at which the line of sight is separated by 2 degrees is defined as VDP. Is VDM. At this time, the central area 161 is within the range defined by the intersection of the line of sight VDP and the display unit 16 and the intersection of the line of sight VDM and the display unit 16. The horizontal range of the display unit 16 is defined by a range in which the line of sight is tilted 2 degrees to the left and a range in which the line of sight is tilted 2 degrees to the right from the center DS0 of the display unit 16. This range is simply expressed as a range of the viewing angle plus or minus 2 degrees from the center of the display unit 16. The gaze icon IC may be displayed in an overlapping manner on the acquired image of the view camera 11 in the central region 161 within the range of the viewing angle plus or minus 2 degrees.

[Structure block diagram]
Next, with reference to FIG. 7, the configuration of the first embodiment will be described with a block diagram. The camera-equipped information terminal 1 according to the first embodiment includes a field-of-view camera 11, an eyeball camera 14, a display unit 16, a computing unit 21, a memory 23, and a communication unit 22. The calculator 21 may include the posture change detection unit 13, the zoom control unit 12, the line-of-sight detection unit 15, and the display control unit 17 as electronic circuits. Alternatively, it may be realized by a process based on a program that executes an equivalent process. The program may be incorporated in the arithmetic unit 21 or may be expanded in the memory 23. The computing unit 21 controls the view camera 11, the eyeball camera 14, the display unit 16, the memory 23, and the communication unit 22, and configures the camera-equipped information terminal 1 according to the first embodiment including these devices. ing.

For example, the computing unit 21 controls the zoom function of the view camera 11 by the zoom control unit 12 or a program corresponding thereto, and enlarges or reduces the image acquired by the view camera 11. This image is displayed on the display unit 16 under the control of the display control unit 17. Further, this image may be stored in the memory 23 or may be transmitted to another device via the communication unit 22. Further, the posture change detection unit 13 can detect the posture change of the view camera 11 from the change of the image of the view camera 11. The detection method will be described later. The change in posture of the view camera 11 also means that the posture of the head of the user of the information terminal 1 with the camera changes.

In addition to the above functions, the display control unit 17 also performs control to display or erase the gaze icon IC in the center of the display unit 16 or in the central region 161 when a predetermined condition is satisfied. The line-of-sight detection unit 15 processes the eyeball image of the user acquired by the eyeball camera 14 to detect the line of sight of the user. The line-of-sight detection is to detect the direction of the line-of-sight or the coordinates where the line-of-sight intersects the surface of the screen of the display unit 16 as the viewpoint VP.

[Processing flow]
Next, a processing flow of the camera-equipped information terminal 1 of the first embodiment will be described with reference to FIG. FIG. 8A is a processing flow for detecting the motion of the user, which has been described with reference to FIGS. 4 and 5. It can be executed by the calculator 21. While the user is using the camera-equipped information terminal 1, this process is repeatedly executed. This processing detects a change in the line of sight of the user and a change in the posture of the view camera 11, and if the directions of both changes are opposite directions, that is, the changes are offset, the gaze icon IC is displayed on the display unit 16 Display at or near the center of. Then, if the user retains the viewpoint VP on the gaze icon IC, the display is maintained, and if the viewpoint VP is not retained, the display of the gaze icon IC is erased.

First, the calculator 21 detects a posture change of the view camera 11 (step STP1), and detects a user's line of sight from an image acquired by the eyeball camera 14 (step STP2). Either step STP1 or step STP2 may be executed first. Next, the process branches based on the condition 1. Condition 1 is'true' if the user's viewpoint VP is outside the central area 161. The condition of the line-of-sight displacement from the vicinity of the center may be that the displacement is larger than the condition 1. However, hereinafter, the fact that the viewpoint VP is outside the central region 161 is also referred to as a “line-of-sight change”. This is because the viewpoint VP of the user is usually inside the central area 161 and the viewpoint VP deviates from the central area 161 according to some movement of the user. If it is not'true', that is, if it is'false', the process is terminated, and if it is'true', the process branches to the subsequent process (step STP3).

Then, the condition 2 is judged. The condition 2 is a judgment based on a comparison between the change of the line of sight, that is, the direction in which the viewpoint VP deviates from the central area 161 and the direction of the posture change of the view camera 11, that is, the posture change of the user's head. In FIGS. 4B1 and 4C above, the viewpoint VP is deviated upward from the central region 161. On the other hand, when the user tilts his/her head forward, the posture of the field-of-view camera 11 is changed downward. In such a case, the change in the line of sight tends to cancel the change in the posture of the camera, and therefore the determination of Condition 2 is “true”. When the camera posture change is not detected or the direction of the camera posture change is not opposite to the direction of the line-of-sight change, it is determined as “false” instead of canceling. Based on the judgment of the condition 2, if "false", the process is ended, and if "true", the process branches to the subsequent process (step STP4).

Then, the gaze icon IC is displayed in the central area 161. This is displayed by being superimposed on the image acquired by the view camera 11. Further, a gaze flag FG, which will be described later, is set. Then, after displaying the gaze icon IC, a predetermined time is left before proceeding to the next process (step STP5). This is because the gaze icon IC is displayed in the central area 161 and the user secures the time for keeping the viewpoint VP there. It is desirable that this predetermined time is within 1 second, but the setting exceeding 1 second is not excluded.

Next, the line of sight is detected again (step STP6), and the condition 3 is judged. In the determination of the condition 3, it is'false' if the user's viewpoint VP is out of the gaze icon IC, and'true' if it is stagnant. It branches based on the judgment of the condition 3 (step STP7). In the case of'false', the gazing flag FG is reset, the gazing icon IC is erased, and the processing is ended (step STP9). On the other hand, in the case of “true”, the process returns to the sight line detection in step STP6 (step STP7). Here, the gaze flag FG is a binary reference flag such as ON and OFF that can be referred to by other processing routines.

To briefly describe the above processing, when the change in the line of sight and the movement of the view camera 11 mounted on the user's head are in a direction canceling each other, the gaze icon IC is displayed in the central area 161. Then, when the user retains the viewpoint VP here, the display of the gaze icon IC is maintained, but unless the viewpoint VP is retained therein, the gaze icon IC is erased. Further, while the display of the gaze icon IC is maintained, the gaze flag FG remains set.

Here, the computing unit 21 is a computing unit that operates in a multithread. That is, a plurality of processing routines are executed in parallel. The arithmetic unit 21 may be either a single processor or a multiprocessor. In the case of a single processor, a plurality of processing routines are executed in a time-sharing manner, and apparently they are executed in parallel. In the case of a multiprocessor, it is assumed that a plurality of processors access a common memory, and the memory access contention is appropriately adjusted.

The process shown in FIG. 8B is performed in parallel with the process described with reference to FIG. Further, it is possible to refer to the gaze flag FG output by the processing described above. Here, the condition 4 is first determined. In Condition 4, if the gaze flag FG is set, it is'true', and if it is reset, it is'false'. If it is'false', the process is terminated, and if it is'true', the process branches to the subsequent process (step STP10). Next, the zoom operation of the view camera 11 is performed via the zoom control unit 12 (step STP11). Then, the condition 4 is judged again (step STP10). By executing this process, the zoom operation of the view camera 11 is continued while the gaze flag FG is set, that is, while the gaze icon IC is displayed in the central area 161.

By the processing flow described above, the user of the camera-equipped information terminal according to the first embodiment can perform the zoom operation of the view camera 11 only by the movements of the head and eyes. If the viewpoint VP continues to stay in the gaze icon IC displayed in the central area 161 near the center of the display unit 16, the set state of the gaze flag FG continues and the zoom operation of the view camera 11 can be continued. Further, in the case where the gaze icon IC is displayed in the central area 161, when the gaze detection unit 15 detects that the visual VP point is out of the gaze icon IC, the gaze icon IC is displayed in step STP9. It is erased, and at the same time, the gaze flag FG is also reset. As a result, the camera zoom operation shown in FIG. 8B also ends.

The jaw pulling operation of the user has already been described with reference to FIGS. 4 and 5. With the jaw-pulling movement, the face on the front of the head tilted downward, while the line of sight changed upward, and the two changed in opposite directions to cancel each other. In contrast to this, the jaw-projecting motion by the user. This will be described with reference to FIG. FIG. 9A1 shows a normal natural state, in which the face direction and the line-of-sight direction match. The direction CD of the view camera 11 mounted on the front surface of the head and the line-of-sight direction CD of the user who defines the viewpoint VP near the center of the display unit 16 match. FIG. 9A2 is an image acquired by the view camera 11, and is also an image displayed on the display unit 16. In this image, the camera line CL and the viewpoint line VL coincide near the center of the display unit 16. The user is gazing at a small white square displayed near the center of the display unit 16.

FIG. 9(B1) shows a state in which the user protrudes his chin while keeping his eyes on the small white square displayed near the center of the display unit 16, and as a result, the face is slightly upward. There is. At this time, the orientation of the view camera 11 changes upward, and the viewpoint VP changes downward. The image acquired by the view camera 11 and displayed on the display unit 16 at this time is shown in (B2). Compared with the image of (A2), it has moved slightly downward. Of course, the camera line CL continues to be located at the center of the display unit 16. On the other hand, the viewpoint line VL is moving downward together with the small white square. At this time, the orientation change direction of the view camera 11 and the orientation change direction of the user are opposite and cancel each other, so that the gaze icon IC is displayed in the central area 161 of the display unit 16 [FIG. C2)].

FIG. 10A1 illustrates a state in which the user keeps the viewpoint VP staying on the gaze icon IC and superimposes a small white square that was initially gazed on the gaze icon IC. (A2) is the display screen of the display unit 16 at this time. 10B and 10C, the zoom operation in which the zoom ratio is reduced is performed around the center of the display unit 16, that is, the gaze icon IC, the object is gradually reduced, and the field of view is gradually enlarged. It shows how it goes.

4 and 5 is an example in which the gaze icon IC is displayed with the direction of the line-of-sight change being upward, and the example illustrated in FIGS. 9 and 10 is that the line-of-sight change is directed downward. This is an example in which the icon IC is displayed. In addition, it is possible to create a case where the line-of-sight change changes to either the left or right direction and the view camera 11 changes its posture in a direction to offset the change. In the second embodiment described below, in the operation of displaying the gaze icon IC, when the sight line change is in the upward direction, the view camera 11 is caused to perform an enlarging zoom operation, and the sight line change is in the downward direction. The field-of-view camera 11 is caused to perform a reduction zoom operation. Here, the enlargement zoom is a zoom operation in which the object becomes larger but the field of view becomes narrower. Then, the reduction zoom is a zoom operation in which the object becomes smaller and the field of view becomes wider.

[Second embodiment Information terminal with camera]
The processing flow of the camera-equipped information terminal 2 according to the second embodiment of the present invention will be described with reference to FIG. The configuration represented by the block diagram of the camera-equipped information terminal 2 is the same as that of the camera-equipped information terminal 1 of the first embodiment. First, reference is made to FIG. The processing from step STP1 to step STP4 is the same as that of the camera-equipped information terminal 1 of the first embodiment. That is, if the direction of the change in the line of sight and the direction of the change in the camera attitude do not cancel each other, the process is terminated. If there is a canceling relationship, the process branches to step STP12.

Condition 5 is that the line-of-sight change detected in the previous step was an upward change. The upward gaze change is a change that occurs when the user pulls the chin and turns the head slightly downward. When the condition 5 is judged and the line-of-sight change is not upward, “false” is determined and the process branches to step STP14. If the line-of-sight change is upward, it is judged as "true" and the process branches to step STP13 (step STP12). In step STP13, a gaze U flag, which will be described later, is set. This is a 1-bit flag that is set when the line of sight change is upward, that is, when the user pulls his chin. This flag is referred to in the processing routine described later with reference to FIG. In step STP14, the gaze D flag is set. This is a 1-bit flag that is set when the gaze change is downward, that is, when the user sticks out his chin. This flag is referred to in the processing routine described later with reference to FIG. However, in the processing flow shown here, the gaze D flag is set for all changes in the line of sight except for the changes in the line of sight. Strictly speaking, if the gaze D flag is set only when the line-of-sight change is downward, it is necessary to branch the step STP12 in three ways. Then, if the change in the line of sight does not correspond to any of the up and down directions, the process may be ended.

In the subsequent processing, the gaze icon IC is displayed in the central area 161 and maintained for a predetermined time (step STP5). This is to secure the time until the user retains the viewpoint VP in the gaze icon IC. Next, the line of sight is detected again (step STP6), and the condition 3 is judged. In the determination of the condition 3, it is'false' if the user's viewpoint VP is not staying in the gaze icon IC, and'true' if it is staying. It branches based on the judgment of the condition 3 (step STP7). In the case of'false', the gaze U flag or gaze D flag is reset, the gaze icon IC is erased, and the process ends (step STP15). On the other hand, in the case of “true”, the process returns to the sight line detection in step STP6 (step STP7). Here, the gaze U flag and the gaze D flag are binary reference flags such as ON and OFF that can be referred to by other processing routines.

To briefly describe the above processing, when the change in the line of sight and the movement of the view camera 11 mounted on the user's head are in a direction canceling each other, the gaze icon IC is displayed in the central area 161. If the user holds the viewpoint VP here, the display of the gaze icon IC is maintained, but if the user does not hold the viewpoint VP there, the gaze icon IC is erased. Also, while the display of the gaze icon IC is maintained, the gaze U flag or the gaze D flag remains set. The gaze U flag is in the set state when the user's action is the action of pulling the chin, and the gaze D flag is in the set state when the user's action is the action of protruding the chin.

The process shown in FIG. 11B is performed in parallel with the process described with reference to FIG. Further, it is possible to refer to the gaze U flag and gaze D flag output in the processing described above. Here, the condition 6 is first determined. In Condition 6, if the gaze U flag is set, it is'true', and if it is reset,'false'. If'false', the process branches to step STP18, and if'true', the process branches to step STP17 (step STP16). In step STP17, the zoom camera of the view camera 11 is performed, and the process returns to step 16 again (step STP17). In step STP18, condition 7 is determined. In Condition 7, if the gaze D flag is set, it is'true', and if it is reset,'false'. If it is'false', the process is terminated. On the other hand, in the case of “true”, the reduction zoom operation of the view camera 11 is performed. Alternatively, the operation of returning to the maximum state of the visual field range may be performed. Then, it returns to step 16 again (step STP19).

As described above, the posture change of the view camera 11 is detected in the downward direction for the user by the processing flow described above. In addition, when it is detected that the viewpoint VP has changed upward from the central area 161, the gaze icon IC is displayed in the central area 161. Then, while the user stays at the viewpoint VP on the gaze icon IC, control for increasing the zoom ratio of the view camera 11 can be performed.

Further, when it is detected that the posture change of the view camera 11 is directed upwards for the user and the viewpoint VP is changed from the central area 161 to the downward direction, the gaze icon IC is displayed in the central area 161. Is displayed. Then, while the user stays at the viewpoint VP on the gaze icon IC, control for reducing the zoom ratio of the view camera 11 can be performed.

Further, when it is detected that the viewpoint VP deviates from the gaze icon IC, the gaze icon IC is erased from the display unit 16. At the same time, the gaze U flag or gaze D flag is also reset, so that the operation of changing the zoom ratio of the view camera 11 ends.

Next, the line-of-sight detection unit 15 will be described with reference to FIG. The line-of-sight detection unit 15 detects the line of sight of the user from the eyeball image acquired by the eyeball camera 14. More specifically, the coordinates of the viewpoint VP on the surface of the display unit 16 where the user's line of sight intersects the surface of the display unit 16 are detected. In the first embodiment and the second embodiment, the position of the pupil of the right eye of the user is detected from the eyeball image. Near infrared illumination can also be applied to the right eye to stabilize pupil detection. In this case, the eyeball camera 14 uses a camera having a high sensitivity in the near infrared region.

Then, prior to the sight line detection, the calibration operation shown in FIGS. 12A1 and 12A2 is required. Calibration points P1, P2, P3, and P4 are displayed at predetermined positions on the display unit 16 of the camera-equipped information terminals 1 and 2 of the first and second embodiments [FIG. 12(A2)]. ]. The user wearing the information terminal tracks these designated calibration points with his/her eyes, and the position of the pupil in the eyeball image at this time is recorded. The purpose of this is to associate the position of the viewpoint VP of the user with the position of the pupil. If this correspondence is made, for example, when the pupil E1 is detected as in (B3) of the same row from the right eye eye image shown in (B2) A), the viewpoint VP is located on the calibration point P1. It is judged that there is. Here, it should be noted that the surface (B1) of the display section 16 visually recognized by the user and the pupil (B3) formed by the image from the eyeball camera are substantially opposite to each other and therefore are laterally reversed. (B3) When the pupil is detected at a position that does not match any of the eyeways, the coordinates of the viewpoint VP are detected by moving inward or outward from the position of the calibration point eyeway in (B1). The above is the processing of the line-of-sight detection unit 15.

Next, the posture change detection unit 13 will be described with reference to FIG. FIG. 13(A2) shows an image obtained by the view camera 11 mounted on the user's head when the view camera 11 is facing the direction CD1 [FIG. 13(A1)]. Here, when the user pulls the chin and tilts the head slightly downward, the direction of the view camera 11 is toward CD2. The image acquired by the camera at this time is shown in FIG. 13(B2). Here, if the optical flow is obtained from the difference between the images of (A2) and (B2), the posture change of the view camera 11 can be detected. Although the optical flow detection method is not limited, for example, a local template pattern is created from the (A2) image [T1, T2, T3 in FIG. 13(C1)], and pattern matching is performed on the (B2) image. Then, the matching area can be obtained as shown in (C2) [M1, M2, M3 of (C2)], and the optical flow is detected as the vector V2 from the position of T2 to the position of M2. The magnitude and direction of this vector are the magnitude and direction of the posture change of the view camera 11. The direction opposite to the direction of this vector is the direction of the posture change of the view camera 11. In this way, the images before and after the change in the posture of the camera can be processed to detect the change and the orientation of the posture of the camera. That is, the posture change detection unit 13 can be realized by the optical flow detection unit. The above is the processing of the posture change detection unit 13. In order to detect a change in the camera posture, an acceleration sensor or a gyro sensor may be attached to the head of the user in addition to detecting the optical flow from the acquired image.

The above is the information terminals 1 and 2 with a camera according to the first and second embodiments. With such a configuration, it is possible to perform the zoom operation of the camera while gazing at the object to be photographed while suppressing an erroneous operation.

[Third Embodiment Information System]
Next, with reference to FIG. 14, an information system 3 according to a third embodiment of the present invention will be described. The information system 3 according to the third embodiment includes a head-mounted terminal 3A and a server 3B. A personal computer can be used as the server 3B. The images obtained by the view camera 11 provided on the head-mounted terminal 3A and the eyeball camera 14 are transmitted to the server 3B. In the server 3B, the image of the view camera 11 is processed to detect the posture change of the view camera 11. In addition, the line-of-sight change is detected by processing the image of the eyeball camera 14.

Then, when the server 3B detects that the posture change of the view camera 11 and the line-of-sight change obtained from the image of the eyeball camera 14 cancel each other, the head-mounted terminal 3A is watched. Send a command to display the icon IC. Upon receiving the command, the head-mounted terminal 3A displays the gaze icon IC in the central area 161 of the display section 16 provided therein. After that, the server 3B obtains the line of sight from the image of the eyeball camera 14, and when it is determined that the viewpoint VP, which is the intersection of the line of sight with the surface of the display unit 16, is out of the gaze icon IC, the head-mounted terminal 3A. , And sends a command to erase the gaze icon IC. On the other hand, when the viewpoint VP stays in the gaze icon IC, the server 3B issues a command to the head-mounted terminal 3A to perform the zoom operation of the view camera 11 to control the zoom ratio to be large or small. send. The head-mounted terminal 3A that has received the command controls the zoom ratio of the view camera 11 to be large or small.

Based on the above operation, FIG. 14A is referred to. The configuration of the head-mounted terminal 3A is obtained by removing the functions of the posture change detection unit and the line-of-sight detection unit from the configuration of the first embodiment shown in FIG. More specifically, the computing unit 31A is the same as the computing unit 21 except that the functions of the posture change detection unit and the line-of-sight detection unit are removed. These processes are performed by the server 3B. With such a configuration, the load on the computing unit 31A is lighter than that on the computing unit 21.

FIG. 14B shows the configuration of the server 3B. The server computing unit 31B includes a line-of-sight detection unit 35, a posture change detection unit 33, a zoom operation command unit 32, and a display control command unit 34. These may be configured by a dedicated electronic circuit, or may be realized as a processing routine for performing software processing of equivalent functions. The posture change detection unit 33 detects the optical flow from the image of the view camera 11 and detects the posture change of the view camera 11. When the head-mounted terminal 3A is provided with an acceleration sensor or a gyro sensor, the output of these devices may be received and the processing may be changed to a process for detecting this optical flow. The line-of-sight detection unit 35 detects the line-of-sight and changes in the line-of-sight from the image of the eyeball camera 14. The display control command unit 34 issues a command to display or delete the gaze icon IC on the display unit 16 to the head-mounted terminal 3A. The zoom operation command unit 32 sends a command to the zoom control unit 12 of the head-mounted terminal 3A to cause the view camera to perform a zoom operation. Communication between the head-mounted terminal 3A and the server 3B is performed via the communication unit 22 and the server communication unit 32. The above is the configuration of the information system 3 according to the third embodiment. With such a configuration, the load on the computing unit 31A of the head-mounted terminal 3A is lightened, and as a result, heat generation of the head-mounted terminal 3A can be suppressed, power consumption can be reduced, and weight reduction can be easily achieved. Has the effect of becoming.

[Fourth Embodiment]
Next, a fourth embodiment according to the present invention will be described with reference to FIG. This embodiment relates to the relationship between the center of the image acquired by the view camera 11 and the center of the display unit 16, and is an embodiment that can be added to all the embodiments from the first embodiment to the third embodiment. FIG. 15A shows the positional relationship between the center IM0 of the image acquired by the view camera 11 and the center DS0 of the display unit 16. The image 113 formed by the lens system 111 of the view camera 11 is converted into an electric signal by the image sensor 112 and transmitted to the display unit 16. Then, this image is displayed on the display unit 16 as an image 16S. At this time, the center IM0 of the image 113 is displayed at the center DS0 of the display unit 16. This presupposes that the user normally places his or her viewpoint on the center DS0 of the display unit 16. Since the center IM0 of the image 113 is coincident with the center DS0 of the display unit 16, when the view camera 11 is zoomed, the display image 16S is enlarged or reduced centering on the portion captured in the center DS0 of the display unit 16. Therefore, in the first to third embodiments, the gaze marker IC is displayed at the center of the display unit 16.

However, there are individual differences among humans, and the line of sight is not always directed to the center DS0 of the display unit 16 in a normal state. Further, the information terminal with camera is not necessarily manufactured by adjusting for each person like the individual eyeglasses. Then, it is expected that the number will be further reduced when the viewpoint is placed on the center DS0 of the display unit 16 in the normal state. Therefore, the fourth embodiment is a configuration in which each user tries to display an image in accordance with the position where the normal viewpoint is placed.

In the information terminal with camera of the fourth embodiment, the coordinate value, which is the position of the viewpoint VP of the user, is acquired as a sample at all times or at regular intervals, and is stored in the memory 23 or the server memory 33. By statistically processing this, the coordinates of the position where the user's viewpoints VP most gather or the coordinates of the position where the user's viewpoints VP gather most are obtained. As the statistical processing, an average value or a mode value can be taken. It is preferable to obtain the sample except the period when the user is displaying the gaze icon IC for the zoom operation. This is because the period in which the viewpoint is guided by the gaze icon IC cannot be said to be the normal viewpoint position of the user. By discarding the old one from the above sample and replacing it with a new one, it follows that the position of the normal viewpoint VP of the user gradually changes.

In the example of FIG. 15B, the coordinates of the mode in which the viewpoint VP is placed are obtained by statistical processing. Here, the coordinates of the mode in which the viewpoint VP is placed will be referred to as the neutral viewpoint position DS1. The image 16Q moved so that the center IM0 of the image 113 coincides with the neutral viewpoint position DS1 is displayed on the display unit 16. (C) is an enlarged view of the main part of the explanation. In the figure, the neutral viewpoint position DS1 appears to move in either the vertical direction on the paper surface with respect to the center position DS0 of the display unit 16, but in reality, it can also move in a direction orthogonal to the paper surface. Then, the central area 161 is also set with the neutral viewpoint position DS1 as the center.

In the fourth embodiment described above, the position corresponding to the average value or mode of the coordinate values representing the position of the viewpoint VP is detected as the neutral viewpoint position DS1, and the image 113 acquired by the view camera 11 at this neutral viewpoint position DS1. The center IM0 of the image is displayed on the display unit 16 so as to match. Further, a central area 161 is provided centered on the neutral viewpoint position DS1.

In order to realize these processes using a program, there is a step of detecting the position corresponding to the average value or mode of the coordinate values representing the position of the viewpoint VP and setting it as the neutral viewpoint position DS1. In the step of displaying the acquired image of the view camera 11 and the gaze icon IC on the unit 16, the center of the image acquired by the view camera 11 is made to coincide with the neutral viewpoint position DS1. Then, the central area 161 is defined around the neutral viewpoint position DS1 and the gaze icon IC is displayed in the central area 161. The arithmetic unit 21 or the arithmetic unit 31B may execute the series of processes described above. By doing so, it is possible to absorb the difference in the way of placing the viewpoint for each user and perform a natural operation for anyone. As described above, the position corresponding to the average value or the mode value of the coordinate values representing the position of the viewpoint VP is detected, and the image 16Q in which the center IM0 of the image 113 acquired by the view camera 11 is matched with this position is displayed on the display unit 16. The information terminal with a camera has been described.

Note that, in FIG. 15, the image 113 acquired by the view camera 11, the display image 16S on the display unit, and the corrected display image 16Q are represented as a thick object. However, this is expressed in this way for convenience of explanation, and since these images are electric signals such as electric charges, they are not actually thick as shown in the figure.

1... Information terminal with camera of the first embodiment, 11... View camera,
111... Lens system of view camera, 112... Image sensor of view camera,
113... Image acquired by the view camera, 12... Zoom control unit,
13... Posture change detection unit, 14... Eyeball camera, 15... Line-of-sight detection unit,
16... Display unit, 17... Display control unit, 1F... Frame,
16A... Display cover, 16B... Mirror, 16C... Imaging prism,
16D... Prism for visual recognition, 16S... Image displayed on display unit,
16Q... Display image after correction on display section, 161... Central area,
2A... control box, 21... arithmetic unit, 22... communication unit,
23... Memory, 2... Information terminal with camera of the second embodiment,
3... Information system of the third embodiment, 3A... Head-mounted terminal,
3B... server, 31A... arithmetic unit, 31B... server arithmetic unit,
32... Zoom operation command unit, 33... Attitude change detection unit,
34... Display control command section, 35... Line-of-sight detection section, CL... Camera line,
VL... viewpoint line, E1, E2, E3, E4... detected pupil,
T1, T2, T3... Template, M1, M2, M3... Matching area,
IC...Gaze icon, FG...Gaze flag,
IM0...Center of image acquired by view camera, DS0...Center of display,
DS1... Neutral viewpoint position, LP1, LP2... Optical path,
P1, P2, P3, P4... Calibration points,
R-Eye... The right eye of the user of the information terminal with camera of the first embodiment,
VD: direction and direction of user's line of sight,
CD, CD1, CD2... View camera direction and direction,
VP... viewpoint, V1, V2, V3... optical flow vector

Claims (8)

A field-of-view camera mounted on the face side of the user facing the direction of the tip of the user's nose,
A posture change detection unit that detects a posture change of the view camera,
An eyeball camera mounted on the face side of the user toward the eyeball of the user,
A line-of-sight detection unit that detects the line of sight of the user from the image acquired by the eyeball camera,
A display unit that is mounted on the head of the user so that the user can visually recognize the image and that displays the image acquired by the view camera; and the posture change detection unit and the line-of-sight detection unit, respectively. When the direction of the detected posture change of the view camera and the direction of the change of the line of sight are offset, within the range of the center of the display unit or the viewing angle plus or minus 2 degrees from the center of the display unit. A gaze icon in a certain central area is displayed overlaid on the image acquired by the view camera,
A camera-equipped information terminal in which a zoom ratio of the view camera is controlled to be large or small while a viewpoint, which is a point where the user's line of sight intersects the surface of the display unit, stays at the gaze icon. The position corresponding to the average value or mode of the coordinate values representing the position of the viewpoint is detected as the neutral viewpoint position, the center of the image acquired by the view camera is made to coincide with the neutral viewpoint position, and the image is displayed on the display unit. The information terminal with a camera according to claim 1, wherein the information is displayed on the display device, and the central area is provided around the neutral viewpoint position. In the case where the attitude change of the view camera is detected in the direction facing downward for the user and the gaze icon is displayed in the central area,
The camera-equipped information terminal according to claim 1 or 2, wherein control is performed to increase the zoom ratio of the view camera while the viewpoint is staying at the gaze icon. In the case where the posture change of the view camera is detected in an upward direction for the user and the gaze icon is displayed in the central area,
The camera-equipped information terminal according to claim 1, wherein control is performed to reduce the zoom ratio of the view camera while the viewpoint is staying at the gaze icon. In the case where the gaze icon is displayed in the central area,
The gaze icon is erased from the central region when it is detected that the viewpoint is out of the gaze icon. An information terminal with a camera according to the item. The posture change detection unit is configured by a gyro sensor mounted on the head of the user, an acceleration sensor, or an optical flow detection unit that detects an optical flow caused by a change in an image of the view camera. The information terminal with camera according to claim 2, claim 3, claim 4, or claim 5. A step of detecting a posture change of a view camera attached to the face side of the user toward the direction toward which the tip of the user's nose is directed;
Detecting the line of sight of the user from an image acquired by an eyeball camera provided in front of the face of the user toward the eyeball of the user,
When the orientation change direction of the view camera and the orientation change direction of the view camera detected in the step of detecting the change in posture of the view camera and the step of detecting the line of sight are offset directions,
The display unit is attached to the user's head so that the user can visually recognize it, and the center of the display unit that displays the image acquired by the view camera, or a range of a viewing angle of ±2 degrees from the center of the display unit. A step of displaying the gaze icon on the display unit by superimposing the gaze icon on the image acquired by the view camera,
While the viewpoint, which is the point where the user's line of sight intersects the surface of the display unit, stays in the gaze icon, a step of controlling the zoom ratio of the view camera to increase or decrease to a computing unit The program to run. A step of detecting the position corresponding to the average value or mode of the coordinate values representing the position of the viewpoint to be the neutral viewpoint position,
In the step of superimposing and displaying the image acquired by the view camera on the display unit and the gaze icon, the center of the image acquired by the view camera is matched with the neutral viewpoint position, and the neutral viewpoint position is set as the center. The program according to claim 7, which defines the central area and causes the computing unit to execute a process of displaying the gaze icon in the central area.