JP6700770B2 - Display device, control method thereof, and control program - Google Patents

Description

  The present invention relates to a display device, a control method thereof, and a control program, and more particularly to a display device in which an image pickup device is arranged on the back surface of a transmissive display panel such as an organic EL panel.

  Generally, in a videophone, conversation is performed by visually observing the face of the other party displayed on the display unit. At this time, the user of the communication device is imaged by the imaging device and the image is sent to the other party. Then, in order to image the user from the front, there is a display device in which a light-transmissive display unit such as an organic EL panel is used as the display unit and the imaging device is arranged on the back side of the display unit.

  By the way, not only is the above-mentioned display device used for a video conference, but also a user's face is photographed by an imaging device, the face image is displayed on the display device, and the display device is used like a mirror. Sometimes. That is, when the communication mode such as a videophone is switched to the mirror mode, the display device displays the face image of the user captured by the imaging device on the display unit of the device itself. At this time, if the eyes of the user and the face image are matched with each other, the user can eliminate the discomfort even when the display device is used as a mirror as in a normal mirror.

  FIG. 6 is a diagram for explaining a situation when applying makeup or the like using a mirror. And FIG.6(a) is a figure which shows the state of the user who faces a mirror, and FIG.6(b) is a figure which shows the image of the user reflected in a mirror.

  An application of the mirror is, for example, facial makeup. When applying makeup, the user facing the mirror sometimes moves his/her face back and forth with respect to the mirror in order to confirm the condition of the makeup (see FIG. 6A). When the user approaches the mirror, the user's face appears large in the mirror, and when the user moves away from the mirror, the user's face appears small in the mirror (see FIG. 6B). That is, when applying makeup to the details of the face, the user brings the face close to the mirror to check the makeup condition. On the other hand, when checking the entire makeup, the user keeps his face away from the mirror. When applying makeup, the user needs to frequently move his/her face back and forth with respect to the mirror.

  On the other hand, when the above-mentioned display device is used as a mirror, it is possible to enlarge and display the face image on the display unit by optical zoom or zoom by digital image processing. However, in order to frequently switch between the normal image and the enlarged image, the user needs to operate operation members such as a start switch for displaying the enlarged image each time, which is extremely complicated for the user. ..

  Therefore, there is a display device in which an image is enlarged according to a gesture of a user without operating the operation member (see Patent Document 1). Here, the display device includes a display, a touch panel, and a camera that captures a hand gesture, and moves, rotates, enlarges, and reduces an image according to the hand gesture.

Japanese Patent Laid-Open No. 11-338120

  However, in Patent Document 1, even when the user unconsciously moves or moves his/her hand by mistake, the motion is determined to be a gesture input, and the image may be moved or rotated. is there.

  Therefore, an object of the present invention is to use a display device as a mirror and control the display magnification by a user's gesture, and the display magnification can be controlled without being affected by a malfunction of the user, and the control thereof. It is to provide a method and a control program.

In order to achieve the above-mentioned object, a display device according to the present invention is a display device including an image capturing unit that captures image data of a subject to obtain image data, and a display panel that displays an image according to the image data. Determination means for detecting the distance between the subject and the display device to determine whether or not the distance is reduced, and detection for detecting whether or not the display device is held by the subject. And the determination means determines that the distance is being reduced, the detection means detects that the display device is being held by the subject, and the image data is detected. And a display control unit for displaying the enlarged image on the display panel by enlarging the enlarged image at a predetermined magnification.

A control method according to the present invention is a control method for a display device, comprising: an image pickup unit that picks up an image of a subject to obtain image data; and a display panel that displays an image according to the image data. A determination step of detecting a distance to the device to determine whether the distance is reduced, a detection step of detecting whether the display device is held by the subject, and the determination If it is determined in the step that the distance is reduced, and if it is detected in the detection step that the display device is held by the subject, an image corresponding to the image data is determined to be predetermined. A display control step of enlarging at a magnification and displaying the enlarged image on the display panel.

A control program according to the present invention is a control program used in a display device having an image pickup means for picking up an image of a subject to obtain image data, and a display panel displaying an image corresponding to the image data. A determining step of determining whether or not the distance between the subject and the display device is reduced by the computer, and whether the display device is being held by the subject. If it is determined that the display device is gripped by the subject in the detection step, if it is determined that the distance is reduced in the detection step and the determination step A display control step of enlarging an image corresponding to the image data at a predetermined magnification and displaying the enlarged image on the display panel is performed.

  According to the present invention, when the display device is used as a mirror and the display magnification is controlled by the gesture of the user, the display magnification can be controlled without being affected by the malfunction of the user.

It is a figure for explaining an example of electronic equipment provided with the display by an embodiment of the invention, (a) is a figure which looked at electronic equipment from the front, and (b) sees electronic equipment with a user from the upper part. It is a figure. It is a block diagram which shows an example about the structure of the portable terminal shown in FIG. 3 is a flowchart (part 1) for explaining an example of a photographing operation by the mobile terminal shown in FIG. 2. 6 is a flowchart (part 2) for explaining an example of a shooting operation by the mobile terminal shown in FIG. 2. FIG. 3A is a diagram for explaining a situation in which the mobile terminal shown in FIG. 2 captures an image of a user in a mirror mode, FIG. And (c) are diagrams showing a state in which an image with a magnification corresponding to the focal length of the camera is displayed. It is a figure for demonstrating the situation at the time of performing makeup etc. using a mirror, (a) is a figure which shows the state of the user who faces a mirror, (b) is a figure which shows the user's image reflected in a mirror Is.

  An example of the display device according to the embodiment of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram for explaining an example of an electronic device including a display device according to an embodiment of the present invention. Then, FIG. 1A is a view of the electronic device viewed from the front, and FIG. 1B is a view of the electronic device viewed from above together with the user. In addition, here, a mobile terminal which is one of the electronic devices will be described as an example.

  The mobile terminal includes a display panel (display unit) 2, and the display panel 2 is, for example, an organic EL panel or the like that can transmit light (that is, includes a light transmission region). An image pickup device (for example, a digital camera) 11 is arranged on the back side of the display panel 2 (+z direction in the figure) so as to face the light transmission area and photograph a user who views the display panel 2. There is.

  FIG. 2 is a block diagram showing an example of the configuration of the mobile terminal shown in FIG.

  With reference to FIGS. 1 and 2, a digital camera (hereinafter simply referred to as a camera) 11 includes a taking lens (not shown), an image sensor 10, and an image sensor drive circuit 51. The camera 11 is positioned at the center of the display panel 2 in the horizontal direction (x direction in the drawing) and in the vertical direction (y direction in the drawing) above the center of the screen (+y direction in the drawing). Take a picture of the user through.

  As shown in FIG. 1A, an operation switch (SW) 40 is arranged on the side surface of the mobile terminal, and the user uses the operation SW 40 to input various commands to the system control circuit 50. For example, the user sets either the communication mode or the mirror mode by the operation SW 40.

  When the communication mode is set, the image sensor drive circuit 51 drives the image sensor 10 under the control of the system control circuit 50. As a result, the image pickup device 10 outputs an analog signal corresponding to the user image (optical image) formed through the taking lens, A/D-converts the analog signal, and outputs it as image data. Output to 51. The image sensor drive circuit 51 sends the image data to the system control circuit 50.

  A frame that is made of a semi-transparent material is disposed around the display panel 2 in a portion facing the user. A microphone 21 and a pair of speakers 31 and 32 are arranged in the frame.

  The voice collected by the microphone 21 is sent to the voice processing circuit 53 as a voice signal. Under the control of the system control circuit 50, the audio processing circuit 53 performs a predetermined process on the audio signal and sends it to the system control circuit 50 as audio data. The system control circuit 50 sends the image data and the audio data obtained by the image pickup device 10 to the mobile terminal of the other party by the transmission/reception circuit 56.

  The transmission/reception circuit 56 is connected to the mobile terminal on the partner side via a network or the like. When the image data and the audio data are received from the mobile terminal of the other party via the transmission/reception circuit 56, the system control circuit 50 causes the display processing circuit 52 to display an image on the display panel 2 according to the image data received from the other party. .. In addition, the system control circuit 50 drives the speakers 31 and 32 by the drive circuit 54, and outputs the audio corresponding to the audio data received from the other party from the speakers 31 and 32.

  The mobile terminal is provided with the acceleration sensor 35, and the sensor drive circuit 55 drives the acceleration sensor 35 under the control of the system control circuit 50. The acceleration sensor 35 detects the period of vibration of the mobile terminal and also detects acceleration in each of the x, y, and z directions. Then, the sensor drive circuit 55 sends the detection result detected by the acceleration sensor 35 to the system control circuit 50.

  The arithmetic circuit 57 performs image processing under the control of the system control circuit 50. The system control circuit 50 also records the image data in the storage circuit 58.

  When the user sets the mirror mode by the operation SW 40, the system control circuit 50 drives the image sensor 10 by the image sensor drive circuit 51 to image the user himself. Then, the system control circuit 50 causes the display processing circuit 52 to display an image according to the image data of the user himself (hereinafter referred to as user image data) on the display panel 2. In the example shown in FIG. 1A, the user's own image is displayed on the display panel 2 in the mirror mode in a horizontally inverted state.

  3 and 4 are flowcharts for explaining an example of the photographing operation by the mobile terminal shown in FIG. The process according to the illustrated flowchart is performed under the control of the system control circuit 50.

  When the power of the mobile terminal is turned on, the mobile terminal is activated, and the system control circuit 50 confirms whether it is the communication mode or the mirror mode depending on the setting state of the operation SW (step S101). Then, the system control circuit 50 determines whether or not the mirror mode is set (step S102).

  If the mirror mode is not set (NO in step S102), that is, if the communication mode is set, the system control circuit 50 confirms the connection state of the transmission/reception circuit 56 with the mobile terminal on the partner side (step S103). Then, the system control circuit 50 determines whether or not the connection by the transmission/reception circuit 56 is completed (step S104). If the connection is not completed (NO in step S104), the system control circuit 50 returns to the process of step S103.

  When the connection is completed (YES in step S104), the system control circuit 50 drives the image sensor 10 by the image sensor drive circuit 51 to photograph the user (step S105). Here, an image including the upper half of the body centered on the user's face is captured. At this time, the system control circuit 50 processes the voice collected by the microphone 21 by the voice processing circuit 53 into voice data.

  Then, the system control circuit 50 causes the transmission/reception circuit 56 to transmit the image data and the audio data to the mobile terminal of the other party. In transmitting the image data and the voice data, the image and the voice data obtained by the mobile terminal are sequentially transmitted. On the other hand, the system control circuit 50 receives the image data and the voice data sent from the mobile terminal of the other side by the transmission/reception circuit 56 as the image data and the voice data of the other side, respectively (step S106).

  The system control circuit 50 causes the display processing circuit 52 to display the image of the other party according to the image data of the other party on the display panel 2 (step S107). At this time, the system control circuit 50 causes the speaker drive circuit 54 to output the sound corresponding to the other party's sound data from the speakers 31 and 32.

  Subsequently, the system control circuit 50 determines whether the power is off (step S108). If the power is not turned off (NO in step S108), the system control circuit 50 returns to the process of step S101 to check the mode. On the other hand, when the power is turned off (YES in step S108), system control circuit 50 determines that the communication has ended, and ends the shooting operation. By the power-off operation, the system control circuit 50 terminates the communication with the mobile terminal of the partner, and terminates the photographing of the user by the camera 11 and the image display of the partner on the display panel 2.

  If the mirror mode is set (YES in step S102), the system control circuit 50 first performs image initial setting (step S110). Here, the system control circuit 50 makes settings relating to the magnification when displaying an image on the display panel 2. For example, the system control circuit 50 sets the flag P related to the magnification of the image. At this time, when displaying an image according to the focal length of the optical system of the camera 11, the flag P=0 is set. On the other hand, when the image is enlarged and displayed at a predetermined magnification (here, about 2 times) so as to have a size suitable for makeup and the like, the flag P=1 is set. Note that the flag P=0 is set in the image initial setting.

  Then, the system control circuit 50 images the user himself/herself with the camera 11 (step S111). That is, the system control circuit 50 drives the image sensor 10 by the image sensor drive circuit 51 to obtain user image data. Then, the system control circuit 50 stores the user image data in the storage circuit 58.

  FIG. 5 is a diagram for explaining a situation in which the mobile terminal shown in FIG. 2 photographs a user in the mirror mode. 5(a) is a diagram showing a situation in which the user moves his/her face forward with respect to the mobile terminal, and FIGS. 5(b) and 5(c) are images with a magnification corresponding to the focal length of the camera. FIG. 7 is a diagram showing a change from a state in which a so-called raw image is displayed (right side view) to a state in which it is enlarged and displayed (left side view).

  In the mirror mode, as shown in FIG. 5, the user moves the face slightly forward with respect to the mobile terminal to check the details of the face. At this time, as described later, the face of the user is enlarged (about twice) and displayed on the display panel 2.

  Next, under the control of the system control circuit 50, the arithmetic circuit 57 determines whether or not the flag P=1 (step S112). If the flag P=1 (YES in step S112), the system control circuit 50 displays an image (enlarged image) enlarged by the display processing circuit 52 at a predetermined magnification (about 2 times) on the display panel 2 ( Step S113). After that, the system control circuit 50 calls the user image data stored in the storage circuit 58, and the arithmetic circuit 57 performs image recognition (step S114).

  On the other hand, if the flag P=0 (NO in step S112), the system control circuit 50 causes the display processing circuit 52 to display the image obtained by the camera 11 (hereinafter referred to as a raw image) on the display panel 2 ( Step S115). After that, the system control circuit 50 proceeds to the process of step S114.

  In the process of step S114, the arithmetic circuit 57 performs image recognition on a plurality of images taken at predetermined time intervals (that is, images obtained in succession). Then, in the present embodiment, under the control of the system control circuit 50, the arithmetic circuit 57 determines whether or not the subject in the image is swinging in the horizontal direction (X direction) at a predetermined speed (step). S116). If the head is in a swinging state (YES in step S116), the system control circuit 50 sets the flag P=0 (step S117). After that, the system control circuit 50 proceeds to the process of step S124 described later.

  Here, when the user performs an action of swinging his/her head to the left and right in a short time, which is generally a negative sense, the arithmetic circuit 57 recognizes the action as an action of denying the situation. Then, by the recognition, the system control circuit 50 ends the image enlargement and displays the raw image on the display panel 2. That is, in the swinging state, the system control circuit 50 sets the flag P=0 and resets the enlarged display.

  In the image recognition, by a known image recognition method, for example, the area of the white eyes located on the left and right of the black eye in the eyeball is detected, and whether the eyeball is directed in the left-right direction according to the ratio of the areas. To detect. Here, since the user is gazing at the display panel 2, the black eye (eyeball) is almost facing the front, and if the head is shaken in that state for a short time, the area of the white eye changes within a predetermined time. The direction of the face is judged by the appearance. In this way, it is possible to determine whether or not the user has swayed his face (neck) left and right by detecting the change in the area of the white eyes located on the left and right of the black eye over time.

  For example, by integrating the changes in the left and right white-eye areas during one cycle, the time when the face faces the right and the time when the face faces the left are obtained. If these integrated areas are equal, the face (neck) is approximated. It can be estimated that the waves are symmetrically shaken. That is, it can be determined that the change in the orientation of the subject is substantially symmetrical.

  If the head is not in the swinging state (NO in step S116), the system control circuit 50 controls the arithmetic circuit 57 to change the size of the subject from the plurality of images taken at predetermined time intervals. Detect changes in size and compare. Then, the arithmetic circuit 57 determines whether or not the face (image) is successively enlarged (step S118).

  When the face (image) becomes larger sequentially (YES in step S118), the system control circuit 50 determines that the distance between the user and the display panel 2 is shortened, and the mobile terminal is detected according to the detection result of the acceleration sensor 35. Vibration is detected (step S119).

  Subsequently, the system control circuit 50 determines whether or not the mobile terminal is in the handheld state based on the vibration detection result (step S120). Usually, the human body is said to periodically oscillate back and forth at about 0.5 to 2 Hz. Therefore, if the detection result of the acceleration sensor 35 is within a predetermined range (for example, about 0.5 to 2 Hz), it is possible to determine that the mobile terminal is in the hand-held state held by the human body.

  If it is in the handheld state (YES in step S120), the system control circuit 50 detects the moving direction of the mobile terminal based on the detection result of the acceleration sensor 35 by the arithmetic circuit 57 (step S121). Then, under the control of the system control circuit 50, the arithmetic circuit 57 determines whether or not the moving direction is the direction of the user (minus Z direction shown in FIG. 1) (step S122).

  If the moving direction of the mobile terminal is the direction of the user (YES in step S122), the system control circuit 50 considers that the image of the user is gradually increasing, and the user grips the mobile terminal. Determine that it is close to the user. Then, the system control circuit 50 determines that the user wants to enlarge the user image and sets the flag P=1 (step S123).

  Subsequently, the system control circuit 50 determines whether or not the mirror mode has ended (step S124). When the mirror mode ends (YES in step S124), system control circuit 50 returns to the process of step S101. On the other hand, if the mirror mode has not ended (NO in step S124), system control circuit 50 returns to the process of step S111. After capturing the latest image of the user in step S111, the flag P is determined in step S112, and since P=1 at present, an enlarged image is displayed in step S113, and the left side of FIG. It becomes the state of the figure.

  It should be noted that in step S118, if the face (image) does not become larger in sequence (NO in step S118), that is, if the size of the face has not changed or has decreased, the system control circuit 50 performs the process of step S124. Proceed to. If the moving direction of the mobile terminal is not the direction of the user (NO in step S122), the system control circuit 50 proceeds to the process of step S124 with the flag P=0. Therefore, after that, even if the process returns to S112, the enlarged image is not displayed and the raw image is displayed.

  If it is not in the handheld state (NO in step S120), the system control circuit 50 determines that the mobile terminal is in a stationary state on a desk or the like. Then, the system control circuit 50 calculates the shake of the image by the arithmetic circuit 57 (step S125). After that, the arithmetic circuit 57 determines whether or not the user who is the subject is shaken based on the shake of the image (step S126). Here, the arithmetic circuit 57 determines by image recognition whether the image shake is a shake caused by a natural shake of the user or whether the user intentionally brings his face close to the mobile terminal.

  As described above, the human body is said to periodically oscillate back and forth at about 0.5 to 2 Hz. Therefore, the arithmetic circuit 57 obtains the shake period of the image, which is the shake period of the subject, and if the shake period is within a predetermined range (about 0.5 to 2 Hz), the subject is the unconscious shake of the user. It is assumed that the image is enlarged due to the blur. That is, the arithmetic circuit 57 does not assume that the user intentionally requested the enlarged image.

  If the image shake is not the subject shake (NO in step S126), the system control circuit 50 proceeds to the process of step S123 in consideration of the fact that the image of the user is gradually increasing, and sets P=1. .. Therefore, when step S113 is reached, an enlarged image is displayed as shown in the diagram on the left side of FIG. On the other hand, if the image shake is subject shake (YES in step S126), that is, if subject shake exists, the system control circuit 50 proceeds to the process of step S124 with the flag P=0. Therefore, the enlarged image is not displayed.

  As described above, in the embodiment of the present invention, it is not necessary to operate the enlarged image activation switch or the like when enlarging the image of the user himself. Then, when the user holds the mobile terminal in hand, the image can be enlarged easily by only bringing the mobile terminal close to the face. That is, the disassembly terminal can be used like a hand mirror.

  Further, in the embodiment of the present invention, the enlarged image is displayed only when the two conditions that the distance between the mobile terminal and the user are close to each other and that the user is not shaken are satisfied. Therefore, it is possible to prevent an erroneous operation in which the enlarged image is displayed due to the hand shake of the user.

  Further, when the mobile terminal is placed on a desk or the like, the image can be easily enlarged only by the user bringing the face close to the mobile terminal. At this time, the enlarged image is displayed only when the two conditions that the distance between the mobile terminal and the user are close and that the user does not unconsciously shake (shake of subject) are satisfied. Therefore, it is possible to prevent an erroneous operation in which a magnified image is displayed due to a user's unconscious shake (object shake).

  As is clear from the above description, in the example shown in FIG. 2, the system control circuit 50 and the arithmetic circuit 57 function as the first determining unit, the second determining unit, and the detecting unit. Further, the system control circuit 50 and the display processing circuit 52 function as a display control means. In the example shown in FIG. 2, at least the image sensor 10, the image sensor drive circuit 51, the system control circuit 50, the arithmetic circuit 57, the display processing circuit 52, the display panel 2, the acceleration sensor 35, and the sensor drive circuit 55 are display devices. Make up.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to these embodiments, and various embodiments within the scope not departing from the gist of the present invention are also included in the present invention. ..

  For example, the function of the above-described embodiment may be used as a control method, and the display apparatus may be caused to execute this control method. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the display device. The control program is recorded in, for example, a computer-readable recording medium.

[Other Embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

2 display panel 10 image sensor 35 acceleration sensor 40 operation SW
50 system control circuit 51 imaging element drive circuit 52 display processing circuit 53 audio processing circuit 56 transmission/reception circuit 57 arithmetic circuit

Claims (9)

A display device comprising: an image pickup means for picking up an image of a subject to obtain image data; and a display panel for displaying an image corresponding to the image data,
First determining means for determining a distance between the subject and the display device to determine whether or not the distance is reduced,
Detection means for detecting whether or not the display device is being held by the subject ,
When the detection unit detects that the display device is being held by the subject when the first determination unit determines that the distance is in a state of being shortened, the image data is detected according to the image data. A display control means for enlarging the enlarged image at a predetermined magnification and displaying the enlarged image on the display panel;
A display device having:   The first determination means determines that the distance is reduced when the subject in the image data is large in a state where the subject is not performing a predetermined motion in the continuously obtained image data. The display device according to claim 1, wherein the display device is a display device.   The display device according to claim 2, wherein the predetermined motion is a motion of swinging the head to the left and right. The display control means, when the display device is moving in the direction of the subject while the display device is being held by the subject, magnifies an image corresponding to the image data at a predetermined magnification, display device according to any one of claims 1 to 3, characterized in that displays the enlarged image on the display panel.   The display device according to claim 2, wherein, when the subject is performing the predetermined operation, the display control unit displays an image corresponding to the image data on the display panel without enlarging the image. When the display device is not gripped by the subject, the image data includes a second determination unit that determines whether or not the subject has a shake with a predetermined cycle,
The display control unit enlarges an image corresponding to the image data by a predetermined magnification and displays the enlarged image on the display panel when the subject does not have a blur by the second determination unit. The display device according to claim 4 , wherein: The display panel has a light transmission region, wherein the imaging means any one of claims 1 to 6, characterized in that it is arranged on a rear surface of the display panel opposite to the light transmitting region Display device according to. A method for controlling a display device, comprising: an image capturing unit that captures image of a subject to obtain image data; and a display panel that displays an image corresponding to the image data.
A determination step of determining whether the distance between the subject and the display device is reduced to detect the distance,
A detection step of detecting whether or not the display device is being held by the subject ;
When it is determined in the determination step that the distance is in a state of being shortened, when it is detected in the detection step that the display device is held by the subject, an image corresponding to the image data is displayed. A display control step of displaying the enlarged image on the display panel by enlarging the image at a predetermined magnification;
A control method comprising: A control program used in a display device, comprising: an image capturing unit that captures image data of a subject to obtain image data; and a display panel that displays an image corresponding to the image data.
In the computer provided in the display device,
A determination step of determining whether the distance between the subject and the display device is reduced to detect the distance,
A detection step of detecting whether or not the display device is being held by the subject ;
When it is determined in the determination step that the distance is in a state of being shortened, when it is detected in the detection step that the display device is held by the subject, an image corresponding to the image data is displayed. A display control step of displaying the enlarged image on the display panel by enlarging the image at a predetermined magnification;
A control program for executing the following.

Images