JP2015119277A - Display apparatus, display method, and display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to display a result of simulation in which a desired shade can be intuitively confirmed.SOLUTION: The display apparatus includes: a position determination part for determining a position of an object in an image and a position of a light source that illuminates the object; a shading change part that generates a shading changed image whose shading of the image of the object is corrected in accordance with a distance from the light source on the basis of a positional relationship between the object and the light source; and a display control part that synthesizes the shading changed image and the photographed image and displays it.

Description

  The present invention relates to a display device capable of displaying a through image on a display screen, a display method, and a display program.

  In recent years, portable devices with a photographing function (photographing devices) such as digital cameras have become widespread. Some types of photographing apparatuses have a display unit and a function of displaying a photographed image. Some display screens display a menu screen to facilitate operation of the photographing device. At the time of shooting, the user can perform shooting and observation operations while confirming the through image displayed on the display unit.

  By the way, in some photographing apparatuses, various settings at the time of photographing are automated. For example, some digital cameras and the like are equipped with an AF function that automates focusing and an automatic exposure (AE) function that automates exposure. However, the automatic exposure controls the exposure of the entire image uniformly by measuring the average amount of reflected light in the shooting range and the amount of reflected light from the subject of interest, but depending on the distance and positional relationship between the light source and the subject. In some cases, the user may not be able to obtain an image of the desired shadow appearance. Even when the exposure is manually adjusted, if the position and distance of the light source with respect to the subject cannot be freely adjusted, it is difficult to photograph the subject with a desired shadow. In the first place, it is difficult to intuitively confirm what kind of shadow is obtained at the position of the subject and the light source.

  Therefore, in Patent Document 1, information that a person is photographed on the screen is received from the camera, and the person's face is extracted from the image based on the information, and the print density is adjusted so that the face has an appropriate density. Techniques for determining are disclosed. By using the technique of Patent Document 1, it is possible to obtain a photograph in which the face portion of a person who is a subject is not crushed dark or flies white.

JP 2004-191699 A

  However, in the technique of Patent Document 1, the face image is determined and the density is optimized by image processing, and the user cannot make the user want a shadow.

  An object of the present invention is to provide a display device, a display method, and a display program capable of displaying a shadow-changed image as observation of how a user desires a shadow, observation for obtaining a captured image, and imaging support display. And

  The display device according to the present invention includes a position determination unit that determines a position of an object in an image and a position of a light source that illuminates the object, and a positional relationship between the object and the light source. A shadow change unit that generates a shadow change image in which the shadow of the image is corrected according to the distance from the light source; and a display control unit that combines and displays the shadow change image on the captured image.

  Further, the display method according to the present invention includes a position determination step in which a captured image of a subject is given from the imaging unit, and the position of the target in the captured image and the position of the light source that illuminates the target are determined, and the target Based on the positional relationship between the object and the light source, a shadow change step for generating a shadow change image in which the shadow of the image of the object is corrected according to the distance from the light source, and the shadow change image are combined with the captured image And a display control step for displaying.

  The display method according to the present invention includes a step of receiving an operation of changing a positional relationship on the screen between a light source displayed on the screen and an object illuminated by the light source, and a sub-image at the time of the position changing operation. And a step of changing the shade obtained on the object according to the positional relationship and displaying it on the sub-image.

  In addition, the display program according to the present invention includes a position determination step for determining a position of an object in the captured image and a position of a light source that illuminates the object when a captured image of the subject is given to the computer from the imaging unit. A shadow change step for generating a shadow change image in which the shadow of the image of the target object is corrected according to the distance from the light source based on the positional relationship between the object and the light source; and the shadow change to the captured image. And a display control step of combining and displaying the images.

  According to the present invention, it is possible to display the result of simulating how the shadow desired by the user can be intuitively displayed.

1 is a block diagram showing a display device according to a first embodiment of the present invention. Explanatory drawing for demonstrating the shadow change image in this Embodiment. The flowchart which shows camera control. Explanatory drawing for demonstrating the production | generation method of a shadow change image. Explanatory drawing for demonstrating the production | generation method of a shadow change image. Explanatory drawing for demonstrating the production | generation method of a shadow change image. The block diagram which shows the 2nd Embodiment of this invention. The flowchart which shows the camera control in 2nd Embodiment. Explanatory drawing for demonstrating the production | generation method of a shadow change image. Explanatory drawing for demonstrating the production | generation method of a shadow change image. Explanatory drawing for demonstrating the production | generation method of a shadow change image. Explanatory drawing for demonstrating the production | generation method of a shadow change image. The flowchart which shows the camera control in a modification. Explanatory drawing for demonstrating the display in a modification. The flowchart employ | adopted as the 3rd Embodiment of this invention. Explanatory drawing for demonstrating the tag information recorded simultaneously with the recording of a picked-up image. Explanatory drawing which shows the example of a display on the display screen 56b.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a display device according to a first embodiment of the present invention. In the present embodiment, the position of the subject that is the target of the shooting support display and the position of the light source that illuminates the subject are detected from the captured image, and the shadow of the target when the position of the target with respect to the light source is changed is detected. By displaying a shadow change image that indicates how to appear, it is possible to effectively support confirmation of a shadow appearance (shadow) desired by the user.

  FIG. 2 is an explanatory diagram for explaining a shadow change image according to the present embodiment. FIG. 2A shows a state where the person 21 holds the casing 24 of the display device with the right hand 22 and takes a selfie while viewing the display screen 12a of the display unit 12 on which the through image is displayed. The person 21 is shooting in a room 25 standing at a position separated from the window 26 by a predetermined distance. FIG. 2B shows a through image 31 displayed on the display screen 12a of FIG. The through image 31 includes a user's confidence image 21a, a user's hand image 22a, a window image 26a, an indoor ceiling image 27a, and a wall image 28a.

  In the example of FIG. 2B, the right side of the face 41 of the person 21 is imaged relatively brightly by the light from the window 26 by the shading display of the image 41 a of the face 41 of the person 21, and the left side of the face 41. Fig. 6 schematically shows a state where an image is taken darker than the right side of the face due to a slight distance from the window 26.

  The right / left light / dark ratio of the face image 41a is considered to be constant even when the distance from the window 26 serving as the light source changes. That is, if there is a difference in brightness between the right and left images of the face image 41a at a position relatively close to the light source corresponding to one step of exposure, the face is positioned even relatively away from the light source. There is a difference in brightness between the right and left images of the image 41a of the portion 41 corresponding to one exposure. However, since the change in the brightness of the background according to the distance from the window (light source) 26 is different from the change in the brightness in the face 41, the face image 41 a when close to the window (light source) 26 is displayed. The impression received by the left and right one-stage brightness change and the impression received by the left and right one-stage brightness change of the face image 41a when far from the window (light source) 26 are significantly different. Normally, the difference between the brightness and darkness of the face image 41a tends to be enlarged as it approaches the window 26, and the brightness and darkness difference of the face image 41a tends to be leveled away from the window 26.

However, it is difficult for general users to predict the positional relationship between the subject and the light source and how the shadow of the subject appears, and if the subject (for example, oneself) is located at any distance from the light source, it is desired. It is not possible to know whether or not a shaded image can be obtained. Therefore, in the present embodiment, when the user changes the position of the subject (for example, his / her standing position), a shadow change image (hereinafter also referred to as a sub-image) indicating what kind of shadow image is obtained. ) Is displayed.
The sub-image is an image obtained by cutting out the object and the light source, and is preferably cut out so that the face and the light source do not protrude in face detection or subject contour determination. A block having a specific size obtained by dividing may be used.

  A shadow display area 43a surrounded by a square in FIG. 2B is a shadow-changed image (sub-image) showing the shadow of the face 41 that will be obtained when the person 21 as the subject approaches the window 26. The shadow display area 44a, which is an area for displaying (image) 43b and is surrounded by a square in FIG. 2B, shows the shadow of the face 41 that would be obtained when the person 21 moved away from the window 26. It is an area | region which displays the shadow change image 44b which shows. In the shade change images 43b and 44b in FIG. 2B, the brightness of the face is shown only in black and white for the sake of simplicity. By displaying such shadow change images 43b and 44b, the person 21 can predict where the subject (for example, himself) stands and what kind of atmosphere the image will be. That is, a step of accepting an operation for changing a positional relationship between a light source displayed on the screen and an object illuminated by the light source on the screen, a step of forming a sub-image at the time of the position change operation, the object There is provided an observation support method including a step of changing the shading obtained above according to the positional relationship and displaying it on the sub-image.

  In FIG. 1, the imaging unit 10 of the display device 23 captures a subject image and outputs a captured image. The captured image from the imaging unit 10 is given to the imaging support unit 11. The imaging support unit 11 includes a face determination unit 11a, and the face determination unit 11a detects a face part in the input captured image.

  The position determination unit 11b determines in which position the subject and the light source that are the target are present in the captured image. For example, if there is a luminance portion of a predetermined level or higher in the captured image, the position determination unit 11b determines that this portion is a light source. In addition, the shadow changing unit 11c determines, for example, the distance distribution of the subject in the screen in order to simulate the state of the subject's shadow according to the distance from the light source.

  The shadow changing unit 11c, when the subject as the object moves in a direction approaching or moving away from the light source position determined by the position determination unit 11b, based on the subject image in the captured image, Each shadow state is obtained by simulation. The shadow changing unit 11c gives a shadow change image, which is a simulation result, to the display control unit 11d. The display control unit 11 d outputs to the display unit 12 a composite image obtained by combining the shadow change image obtained by changing the shadow of the subject according to the simulation result with the captured image from the imaging unit 10.

  The display unit 12 is provided, for example, in a housing that houses the imaging unit 10, and the user can visually recognize the display unit 12 during a shooting operation of the imaging unit 10. On the display screen 12 a of the display unit 12, the shadow change image is displayed so as to be superimposed on the through image. By referring to the shadow change image, the user can move the subject in any direction with respect to the light source. It is possible to determine whether or not a captured image of a desired shadow can be obtained. Here, a familiar camera, a smartphone, or the like is assumed, but of course, the display unit 12 may be provided remotely to perform a shadow simulation in a remote place.

  Next, an embodiment configured as described above will be described with reference to FIGS. FIG. 3 is a flowchart showing camera control, and FIGS. 4 to 6 are explanatory diagrams for explaining a method of generating a shadow change image. In FIGS. 3 to 6, description will be made on the assumption that the target for photographing support display is a human face.

  The imaging support unit 11 determines whether an imaging mode is instructed in step S11 of FIG. When the shooting mode is not instructed, the shooting support unit 11 shifts to another mode.

  When the shooting mode is instructed, in step S12, the shooting support unit 11 captures a captured image from the imaging unit 10, and the face determination unit 11a determines a face portion in the captured image. The shadow changing unit 11c determines the distance distribution in the screen for each part in the captured image. The display control unit 11d of the imaging support unit 11 performs predetermined image processing on the captured image from the imaging unit 10, and provides the captured image after the image processing to the display unit 12 to display a through image.

  Next, the shadow changing unit 11c determines, for example, whether or not a face image having a sufficiently large shadow difference exists in the screen (step S13). For example, when a face having a sufficiently large shadow difference exists in the center of the captured image, whether or not there is a light source that illuminates such a face as an object is stabilized by the position determination unit 11b (Step S11). S14).

  Note that the light source position determination in step S14 assumes a light source that emits diffused light, such as a window or a luminaire, and does not assume that the subject is illuminated by direct sunlight. For example, when the brightness difference between the bright and dark parts of the subject is extremely large, it may be determined that the subject is exposed to direct sunlight. In this case, even if the subject moves in the direction of the light source or in the opposite direction, the change in brightness of the captured image remains large and does not change, so advice such as the impression of the captured image changes greatly depending on the face orientation is displayed. You may make it transfer a process from step S14 to step S21.

  If the light source position cannot be determined, the position determination unit 11b may display a warning display indicating that the light source position cannot be determined by the display control unit 11d. For example, the display control unit 11d displays on the display screen 12a of the display unit 12 a warning display such as “The light source position could not be determined”.

  When capturing an image of a room with a relatively dark background and a bright window, it is considered that the brightness of the captured image is dominated by the lighting effect from the window. The light that illuminates the face of a person in the room also depends on the light beam diffused from the window. When the window is simplified as a light source, the brightness of the room is considered to be inversely proportional to the square of the distance from the window.

  Therefore, the shadow changing unit 11c considers the influence of the diffused light from the window, and when there is a light source that illuminates the object in order to simulate the shadow state of the object when the position is moved. Then, it is determined whether or not the distance from the light source to the face part that is the object can be determined (step S15). When the distance can be determined, the shadow changing unit 11c generates a shadow changed image in steps S16 and S17. The shadow changing unit 11c determines the position where the shadow changed image is displayed, simulates the shadow when the object is present at the position, and reflects it in the shadow changed image.

  FIG. 4 shows a state where a subject is photographed as viewed from above, and is for explaining a method of analyzing a shadow state due to the influence of the distance from the light source and the brightness. FIG. 4 shows an example in which the shadow change image is displayed at a position of ½ and 2/3 of the distance between the light source and the object with reference to the position of the light source. 4A shows a state in which a window 26 as a light source is positioned beside a person 21 as a subject, and the person 21 as a subject is illuminated from the side by scattered light from the window 26. A state in which the person 21 as the subject is photographed from directly in front of the person 21 is shown.

  The window 26 emits scattered light from the radiation surface facing the room 25 toward the room 25. Behind the person 21 is a wall 28 as a background. The person 21 is located in front of the wall 28 at a position where the scattered light from the window 26 is received from the side. The direction orthogonal to the radiation surface of the window 26 is defined as the X direction. The arrow from the window 26 in FIG. 4A indicates irradiation of scattered light into the room 25. The fan-shaped broken line in FIG. 4A schematically shows how scattered light diffuses, and the scattered light from the window 26 becomes gradually darker in inverse proportion to the square of the distance as the distance from the window 26 increases. It is assumed that the distance from the imaging unit 10 to the person 21 of the display device 23 is Df, and the distance from the imaging unit 10 to the window 26 that is a light source is Dw. Further, the distance between the subject person 21 and the window 26 is assumed to be Xwf.

  FIG. 4B shows a change in brightness according to the distance from the window 26 that is a light source, for example, at the height of the face (a predetermined horizontal line including the face in the captured image). . It is assumed that the brightness of the face of the person 21 whose distance from the window 26 is Xwf is Bf. In this case, the brightness at the position where the distance from the window 26 is Xwf / 2 is 4Bf, and the brightness at the position where the distance from the window 26 is 3Xwf / 2 is 9/4 (about Bf / 2). . For example, in the case where the shadow changing image is arranged at a position where the distance from the window 26 is Xwf / 2, the shadow changing unit 11c has a shadow obtained by setting the brightness of the face in the captured image to ½ of the original image. Generate a modified image.

  As described above, the shadow changing unit 11c changes the brightness of the shadow change image according to the display position of the shadow change image. However, the display position of the shadow change image is obtained based on the position of the window 26, and when the window 26 occupies a relatively large area in the captured image, it is difficult to determine the reference of the display position of the shadow change image. Become. Therefore, in the present embodiment, the shadow changing unit 11c determines the display position of the shadow changed image based on the angle of view.

  FIG. 5 shows how to obtain the display position of the shadow change image. FIG. 5A shows a photographing state similar to that in FIG. FIG. 5A shows an example in which the person 21 as the object is located in the center of the imaging range, the maximum angle of view is θ0, and the window 26 as the light source is detected at the angle of view θw by a window frame or the like. Yes. If the angle of view at the position where the distance from the window 26 is Xwf / 2 is θl, θl is given by the following equation (1).

θl = {arctan (Xwf / 2)} / Df (1)
FIG. 5B shows the display position of the shadow change image based on the above equation (1). FIG. 5A shows a shooting support display on the display screen 12a of the display unit 12, and the shooting support display is a display of a shadow change image on a through image. The shadow display area 43a on the light source side is displayed at the position of the field angle θl. Further, as shown in FIG. 5B, the position where the distance from the window 26 is 3Xwf / 2 is represented by an angle of view θd having the same size as the angle of view θl and a symmetric position with respect to the center of the screen. .

  FIG. 6 illustrates the image signal level of the shadow-changed image, with the horizontal position of the screen on the horizontal axis and the image signal level on the vertical axis. 6A shows the image signal level of a predetermined line including the face portion of the captured image, FIG. 6B shows the image signal level of the shadow change image arranged and displayed in the light source direction, and FIG. Indicates the image signal level of the shade-changed image arranged and displayed in the direction opposite to the light source direction.

  FIG. 6 (b) is obtained by doubling the face image signal amplitude of FIG. 6 (a), and FIG. 6 (c) is ½ times the face image signal amplitude of FIG. 6 (a). Is. The shade-changed image arranged at the light source side display position in FIG. 5B has a brighter image than the background because the amplitude of the image signal of the original face image is doubled. For example, an image having an impression that the face is whiter than the image of the face in the current captured image is obtained. Further, since the amplitude of the image signal of the original face image is halved in the shadow change image arranged in the direction opposite to the light source direction, an image having a darker impression than the background can be obtained. For example, even when the face portion of the captured image is whiteout, the image is appropriately exposed.

  In step S16, the shadow changing unit 11c generates, as a shadow changed image, an image in which, for example, the signal amplitude of the image signal of the face determined by the face determining unit 11a is enhanced twice. The display control unit 11d synthesizes the captured image and the shade-changed image so that the generated shade-changed image is positioned at the angle of view θl (FIG. 5B) in the light source direction from the face in the captured image. To the display unit 12. Thereby, for example, the shadow change image 43b in FIG. 2B is displayed at the position of the angle of view θl.

  In step S17, the shadow changing unit 11c generates, as a shadow changed image, an image in which the signal amplitude of the face image signal determined by the face determining unit 11a is reduced by, for example, ½. The display control unit 11d displays the captured image, the shadow-changed image, and the shadow-changed image so that the generated shadow-changed image is positioned at the angle of view θd (FIG. 5B) in the opposite direction of the light source from the face in the captured image. Are combined and given to the display unit 12. Thereby, for example, the shadow change image 44b in FIG. 2B is displayed at the position of the angle of view θd.

  The shadow changing unit 11c has shown an example in which the amplitude of the image signal is doubled or halved and displayed at the positions of the view angles θl and θd. However, how many times the amplitude is to be multiplied can be set as appropriate. Yes, the angle-of-view-change image may be displayed at the display position corresponding to the shadow enhancement amount. Not only the method of changing the amplitude of the face portion of the image signal, but also a method of changing the brightness of the face portion of the image signal so as to become brighter or darker as a whole may be adopted.

  In step S <b> 18, the imaging support unit 11 determines whether the user has selected one of the shadow change images displayed in the light source direction or the reverse direction of the light source. When the user performs an operation of selecting one of the shadow change images, a guide display that prompts the user to move the subject in the selected direction is displayed (step S19). In step S20, the display control unit 11d deletes the display of the shadow change image.

  In step S <b> 21, the display device 23 determines whether or not a release operation or a touch operation for performing shooting has been performed. If there is a release operation or the like, the display device 23 performs shooting and records a captured image (step S22).

  Thus, in the present embodiment, a shadow change image is created by simulating the subject image when the subject moves in the light source direction and when the subject moves in the opposite direction from the light source, and the shadow enhancement amount of the shadow change image Is displayed at the display position corresponding to. This shade-changed image indicates where the subject moves and what kind of shade (shading) is taken, and can effectively assist the user in obtaining the desired shaded image. it can. For example, if the user wants to shoot with a whiter skin, wants to shoot with less wrinkles, or wants to shoot without overexposure, it indicates where the subject should stand be able to. By referring to the image state and display position of the shadow change image, the user can easily grasp where the subject should be positioned in order to obtain a desired shaded captured image.

  In the above description, in order to simplify the description, the case where the light source is positioned in the horizontal direction of the target object has been described. However, the light source may be positioned at any position other than right behind or in front of the target object. , As well as applicable.

(Second Embodiment)
FIG. 7 is a block diagram showing a second embodiment of the present invention. This embodiment is applied to a digital camera.

  In FIG. 7, the display device 50 according to the present embodiment is provided with a photographing optical system 61 on the front surface. The imaging optical system 61 guides the subject image to the imaging unit 52. Note that the photographic optical system 61 may be configured separately from the display device 50 and an interchangeable lens that is detachably attached.

  The photographing optical system 61 includes a lens (not shown) for zooming and focusing. These lenses are driven and controlled by a lens control unit 62. The lens control unit 62 is controlled by a signal processing and control unit 53 described later. In the photographing optical system 61, the focus position, the zoom position, the aperture, and the like are controlled by the signal processing and control unit 53.

  The display device 50 includes an imaging unit 52 configured by an imaging element such as a CCD or a CMOS sensor. The imaging unit 52 photoelectrically converts the subject image from the imaging optical system 61 to obtain a captured image. The imaging unit 52 is driven and controlled by the signal processing and control unit 53 and outputs a captured image.

  The signal processing and control unit 53 includes, for example, a CPU and a memory (not shown), and performs camera control according to a program stored in the memory. The signal processing and control unit 53 outputs a drive signal of the image sensor to the imaging unit 52 to control the shutter speed, the exposure time, and the like, and reads a captured image from the imaging unit 52. The signal processing and control unit 53 performs predetermined signal processing, for example, color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing on the read captured image.

  The display device 50 is also provided with an operation determination unit 54 and a clock unit 55. The clock unit 55 generates time information used by the signal processing and control unit 53. The operation determination unit 54 generates an operation signal based on a user operation to an operation unit including a shutter button, a function button, various switches such as a shooting mode setting (not shown) provided in the display device 50, and a signal processing and control unit 53 is output. The signal processing and control unit 53 controls each unit based on the operation signal.

  The signal processing and control unit 53 can perform processing related to recording and reproduction of a captured image. For example, the signal processing and control unit 53 can compress the captured image after the signal processing, and can apply the compressed image to the recording unit 57 for recording. For example, a card interface can be adopted as the recording unit 57, and the recording unit 57 can record image information, audio information, and the like on a recording medium such as a memory card.

  The display control unit 53a of the signal processing and control unit 53 executes various processes related to display. The display control unit 53 a can give the captured image after the signal processing to the display unit 56. The display unit 56 has a display screen such as an LCD, and displays an image given from the display control unit 53a. Further, the display control unit 53a can display various menu displays and the like on the display screen 56b (see FIG. 10A) of the display unit 56.

  The signal processing and control unit 53 can read out the captured image recorded in the recording unit 57 and perform expansion processing. The display control unit 53a can reproduce the recorded image by giving the expanded captured image to the display unit 56.

  The display unit 56 includes a display screen 56b (see FIG. 10A) that displays a captured image, and a touch panel 56a is provided on the display screen 56b. The touch panel 56a can generate an operation signal corresponding to the position on the display screen 56b pointed by the user with a finger. This operation signal is supplied to the signal processing and control unit 53. Accordingly, when the user touches or slides on the display screen 56b, the signal processing and control unit 53 causes the user's touch position, an operation for closing and separating the finger (pinch operation), a slide operation or a slide operation. Thus, it is possible to detect various operations such as the position reached, the slide direction, the touch period, and the like, and it is possible to execute processing corresponding to the user operation.

  The display unit 56 is disposed so as to occupy substantially the entire rear surface of the display device 50, for example, and the photographer confirms the through image displayed on the display screen 56b of the display unit 56 at the time of shooting. It is also possible to perform shooting operations while checking through images.

  The display device 50 is provided with an acceleration determination unit 58. The acceleration determination unit 58 includes a gyro sensor, an acceleration sensor, and the like, and can detect the movement and posture (tilt) of the housing that houses the display device 50. The determination result of the acceleration determination unit 58 is given to the signal processing and control unit 53.

  The display device 50 is also provided with a communication unit 59. The communication unit 59 communicates with an external device to transmit a captured image from the signal processing and control unit 53 and to output a captured image received from the external device to the signal processing and control unit 53. It can be done.

  In the present embodiment, the signal processing and control unit 53 uses the shadow change image to indicate in which position the subject should be positioned with respect to the light source in order to obtain a shaded image desired by the user. The mode can be executed. In order to generate the shadow change image, the signal processing and control unit 53 includes a face detection unit 53b, a shadow change unit 53c, a distance distribution determination unit 53d, and a light source position determination unit 53e in addition to the display control unit 53a. Yes.

  The face detection unit 53b can determine a human face, a human body, a pet, a specific object, or the like in the captured image as a target of the shadow change image by a known face detection process for the captured image. Furthermore, the face detection unit 53b may specify the center face of the screen, the center of the screen, or the like as an object.

  The distance distribution determination unit 53d calculates a distribution of distances from the imaging unit 52 to each unit in the captured image. For example, the distance distribution determination unit 53d can determine the distance from the imaging unit 52 to the subject, the distance from the imaging unit 52 to the light source, and the like. The distance distribution determination unit 53d may perform distance determination using a dedicated distance sensor, may perform distance detection by an image plane phase difference method using the focus control pixels of the imaging unit 52, and may be configured to perform photographing optical. The distance determination may be performed based on the lens position when the system 61 is in focus.

  In the present embodiment, the light source position determination unit 53e determines in which position the light source is located with respect to the subject in the captured image based on the determination result of the distance distribution determination unit 53d. For example, if there is a luminance portion of a predetermined level or higher in the captured image, the position determination unit 11b determines that this portion is a light source.

  The shadow changing unit 53c determines the subject in each case based on the subject image in the captured image when the subject as the target moves in a direction approaching or moving away from the light source position determined by the position determination unit 11b. Each shadow state is obtained by simulation. The shadow changing unit 53c receives the determination result of the distance distribution of the subject in the screen from the distance distribution determining unit 53d, and simulates the shadow state of the subject according to the distance from the light source to generate a shadow changed image. .

  The shadow changing unit 53c gives the display control unit 53a information on the shadow changed image and its display position, which are simulation results. The display control unit 53a generates a composite image in which the shadow change image obtained by changing the shadow of the subject according to the simulation result is combined with the captured image from the image pickup unit 52 at the display position specified by the shadow change unit 53c, and the display unit 56 Output to.

  The display unit 56 is provided, for example, in a housing that houses the imaging unit 52, and the user can visually recognize the display unit 56 during a shooting operation of the imaging unit 52. On the display screen 56b of the display unit 56, the shadow change image is displayed so as to be superimposed on the through image. By referring to the shadow change image, the user can move the subject in any direction with respect to the light source. It is possible to determine whether a captured image of a desired shadow can be obtained.

  In the present embodiment, the shadow changing unit 53c can receive a user operation on the touch panel 56a and can instruct the user of the position where the shadow changed image is displayed. For example, when the user touches the position corresponding to the object on the touch panel 56a and performs a slide operation to a predetermined position, the object moves to an actual position corresponding to the position of the end point of the slide operation. A shadow change image as a simulation result is displayed at a position on the screen corresponding to the position.

  It should be noted that the shadow changing unit 53c can perform a simulation calculation even during the user's slide operation, and the user confirms that the display position of the shadow change image changes and the state of the shadow changes according to the slide operation. However, a slide operation can be performed.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 8, 9, and 10A to 10C. FIG. 8 is a flowchart showing camera control in the second embodiment. In FIG. 8, the same steps as those in FIG. 9 and 10A to 10C are explanatory diagrams for explaining a method of generating a shadow change image.

  In the first embodiment, an example has been shown in which light from a light source (window) illuminates a face from the side. The simulation of the shadow was made easy by assuming that the light hits from the side of the face. On the other hand, in the present embodiment, an example is shown in which light from the light source strikes the face obliquely. In this case, when the moving direction of the target face is not the light source direction, the shadow pattern changes due to the unevenness of the face. Therefore, in order to generate an accurate shade-changed image, a complicated calculation using the facial concavo-convex pattern is required. Therefore, in the present embodiment, in order to simplify the simulation calculation of the shadow change image, the face is assumed to move in the light source direction or the opposite direction.

  Further, in the following description, in order to simplify the description, it is assumed that the light source and the object are positioned at the same height, considering only the X direction (horizontal direction) and the Z direction (depth direction). However, when the heights of the light source and the object are different, a three-dimensional calculation may be performed in consideration of the Y direction (height direction).

  In step S31 of FIG. 8, the signal processing and control unit 53 determines whether or not the shadow simulation mode is designated. When the shadow simulation mode is designated, the light source position determination unit 53e and the distance distribution determination unit 53d determine the position of the light source and the target object such as a face.

  As shown in FIG. 9A, a lamp 72 as a light source is positioned on the left side of a person 71 standing in a room 70, and the person 71 as a subject is illuminated from the left front side by the lamp 72. Shall. FIG. 9B shows the positional relationship when the person 71 as the subject is photographed by the display device 50 as viewed from above. FIG. 9C is for explaining the incidence of light on the person 71 based on the positional relationship between the lamp 72 and the person 71.

  FIG. 9A shows a display device in a state in which a lamp 72 is positioned in front of the subject person 71 diagonally to the left and the subject person 71 is illuminated from the front left by scattered light from the lamp 72. Reference numeral 50 denotes a state in which the person 71 as a subject is photographed from the front of the person 71. The lamp 72 emits scattered light toward each part of the room 70. An arrow from the lamp 72 in FIG. 9B indicates irradiation of scattered light toward the person 71. The fan-shaped broken line in FIG. 9B schematically shows how scattered light diffuses in the direction of the person 71, and shows that the scattered light from the lamp 72 gradually becomes darker as it moves away from the lamp 72. Yes. It is assumed that the distance from the imaging unit 52 of the display device 50 to the person 71 is Df, and the distance from the imaging unit 52 to the lamp 72 is Dl. These distances are obtained by the distance distribution determination unit 53d. In addition, the angle of view of the lamp 72 is θl with reference to the direction of the person 71 (optical axis direction).

The distance distribution determination unit 53d calculates a distance Dlf from the lamp 72 to the person 71. The distance between the lamp 72 and the person 71 in the direction perpendicular to the optical axis of the imaging unit 52 (X direction) is represented by Dl tan θl. Therefore, the distance Dlf is expressed by Dlf = {(Dltanθl) ² + (Df−Dl) ² } ^0.5 .

  If the light source and face position cannot be determined, the process proceeds from step S32 to step S38, a warning display indicating that the light source and face position cannot be determined is displayed, and shadow simulation is performed. Exit mode. Next, in step S33, the shadow changing unit 53c determines whether or not an operation of touching and sliding the face that is the object has been performed in order to display the shadow changed image. If no touch or slide operation is performed, the process proceeds to step S21.

  When the touch and slide operations are performed, in step S34, the shadow changing unit 53c copies the image of the face that is the object, and starts generating a shadow changed image. In the present embodiment, the display position of the shadow change image and the state of the shadow corresponding to the display position are calculated in accordance with an operation in which the user touches and slides the subject in the through image on the touch panel 56a. It has become. In this case, it is considered that the user is not aware of the direction of the straight line 74 that connects the lamp 72 and the person 71 as the subject. Therefore, in the present embodiment, the display position and the shadow state corresponding to the amount of movement of the slide operation are obtained on the assumption that the user slides on the touch panel 56a in the left-right direction (X direction).

  FIG. 9C is for explaining the influence of the shadow due to the movement, assuming that the person 71 moves on a straight line 74 connecting the lamp 72 and the person 71, and the positions of the person 71 and the lamp 72 on the coordinates. Is displayed. In FIG. 9C, the coordinates of the person 71 are (0, Df), and the coordinates of the lamp 72 are (Dltan θl, Dl).

  Therefore, when the slide operation is performed in the X direction, if the actual moving distance corresponding to the sliding operation is X1, the moving distance of the person 71 is X1 · Dlf / (Dltanθl). The brightness of the shadow change image may be controlled in inverse proportion to the square of the moving distance. In addition, since it changes linearly in the direction of the straight line 74 with respect to the change in the X direction, the brightness of the shadow change image may be controlled in inverse proportion to the square of the change rate in the X direction.

Further, the actual X-direction movement amount X with respect to the horizontal slide amount Xp on the touch panel 56a is ½ of the horizontal width of the touch panel 56a because the actual X-direction distance corresponding to the maximum angle of view θ0 is Dftan θ0. If the distance is Xpo, it can be obtained by the following equation (2).
X = Df (tan θ0) Xp / Xp0 (2)
In this embodiment, in order to indicate that the shadow change image is moved along the straight line 74 by an operation in the X direction by the user, the size of the person 71 in the shadow change image is set to the movement amount in the Z direction. It is designed to change accordingly.

  Since the angle between the X direction and the straight line 74 is (π / 2) −θl, the change amount Z in the Z direction is given by the following equation (3), where X is the change in the X direction (step S35).

Z = −tan {(π / 2) −θl} X + Df (3)
If the shadow change image is moved on the straight line 74, the size of the face in the shadow change image changes at a ratio of Z and Df. In step S36, the shadow changing unit 53c changes the size of the face image that is the copied object based on the ratio of Z and Df (step S36).

  Next, the shadow changing unit 53c generates a shadow changed image in which the shadow of the face image that is the copied object is emphasized or weakened based on the change rate in the X direction corresponding to the slide amount (step S37). The shadow change unit 53c gives the generated shadow change image to the display control unit 53a. The display control unit 53a combines the live view image and the shadow change image so that the shadow change image is displayed at a position corresponding to the slide operation. This composite image is displayed on the display screen 56 b of the display unit 56.

  10A to 10C are diagrams for explaining screen display in the shadow simulation mode. FIG. 10A shows a state in which a through image 81 is displayed on the display screen 56 b, and images of the lamp 72 and the person 71 that are light sources are displayed in the through image 81. The user is about to touch the face of the person 71 as the object with the fingertip 82. By this touch operation, the face image of the person 71 is copied by the shadow changing unit 53c.

  FIG. 10B shows that the user has slid the fingertip 82 to the right. The shadow changing unit 53 c detects the user's slide operation in the direction of the arrow 83, and displays the shadow changed image 84 at the position of the fingertip 82. The user desires to display a shadow change image when the user moves the person 71 in a direction approaching the lamp 72. Since the lamp 72 is positioned on the near side of the person 71, the size of the shadow change image 84 is determined. Is displayed larger than the original face image. As a result, the user can recognize that the object has moved to the near side based on the image size, and can know the state of the shadow in this case from the shadow of the shadow change image 84.

  FIG. 10C shows that the user has slid the fingertip 82 to the left. The shadow changing unit 53 c detects the user's slide operation in the direction of the arrow 85 and displays the shadow changed image 86 at the position of the fingertip 82. When the user moves the person 71 in the direction away from the lamp 72, the user desires to display the shadow change image, and the size of the shadow change image 86 is displayed smaller than the original face image. Thus, the user can recognize that the object has moved in the depth direction based on the image size, and can know the state of the shadow in this case from the shadow of the shadow change image 86.

  Thus, also in this embodiment, the same effect as the first embodiment can be obtained. Further, in the present embodiment, even when illumination from a light source or an object is irradiated obliquely, a simple calculation is performed by simulating a shadow when the object is moved in the direction of the light source or in the opposite direction. Thus, it is possible to generate a shadow change image as a simulation result. Further, in the present embodiment, the user can instruct the position where the shadow change image is displayed on the touch panel, the shadow state at the position in accordance with this instruction is calculated, and the shadow is changed to the instructed position. The image can be displayed, and the user can easily confirm the position of the object from which the desired shadow can be obtained.

  In the above description, in order to show the change in the distance from the imaging unit, the size of the face in the shade-changed image approaching the light source direction is increased, and the size of the face in the shadow-changed image moved in the opposite direction of the light source However, the size of the face may or may not match the actual size, or may be displayed in the same size.

(Modification)
11 and 12 show a modification. FIG. 11 is a flowchart showing camera control in the modified example, and FIG. 12 is an explanatory diagram for explaining display in the modified example. In FIG. 11, the same steps as those in FIG.

  In this modification, a shadow change image is displayed when the light source is moved instead of the object. If it is determined in step S33 in FIG. 11 that there is no face slide operation, the shadow changing unit 53c determines in step S41 whether there has been a light source touch and slide operation.

  When the touch and the slide operation are performed on the light source, the shadow changing unit 53c copies the image of the light source and starts generating the shadow changed image in step S42. Also in the present modification, the display position of the shadow change image and the state of the shadow corresponding to the display position are calculated in accordance with an operation in which the user touches and slides the light source in the through image on the touch panel 56a. It has become.

  Also in this case, for example, it is assumed that the lamp 72 moves on a straight line 74 connecting the lamp 72 and the person 71 in FIG. The shadow changing unit 53c calculates the amount of movement of the lamp 72 in the Z direction based on the actual distance in the X direction based on the operation of touching and sliding the lamp 72 (step S43). The size of the lamp 72 changes with the ratio of Z and Df. In step S44, the shadow changing unit 53c changes the size of the image of the lamp 72, which is the copied light source, based on the ratio of Z and Df.

  Further, in this modification, when the movement position of the lamp 72 that is a light source is designated, the shadow changing unit 53c determines the face portion of the person that is the object based on the positional relationship between the lamp position and the object. The shadow is simulated and the shadow change image, which is the simulation result, is displayed so as to overlap the display position of the original object.

  FIG. 12 is a view for explaining screen display in the shadow simulation mode. FIG. 12A shows a state in which a through image 81 is displayed on the display screen 56 b, and images of the lamp 72 and the person 71 that are light sources are displayed in the through image 81. The user is about to touch the lamp 72 as a light source with the fingertip 82. By this touch operation, the image of the lamp 72 is copied by the shadow changing unit 53c.

  FIG. 12B shows that the user has slid the fingertip 82 to the left. The shadow changing unit 53 c detects the user's sliding operation in the direction of the arrow 91 and displays a lamp image 92 at the position of the fingertip 82. The user desires to display a shadow-changed image when the user moves the lamp 72 in a direction approaching the object. Since the lamp 72 is positioned in front of the person 71, the size of the lamp image 92 is determined. Is displayed smaller than the image of the original lamp 72. Thereby, the user can recognize that he is going to display the shadow change image when the light source is moved in the depth direction. In step S37, the state of the shadow in this case is displayed by the shadow change image. The shadow change image in this case is obtained by changing the image portion of the face portion of the original person 71 in the through image to a shadow based on the simulation result.

  As described above, in this modified example, even when the light source is moved, it is possible to simulate the shadow when the light source is moved and display the shadow changed image. Needless to say, it is not necessary to directly touch with a finger, and it goes without saying that various methods can be used to change the position, such as specifying an object (light source or subject) with voice or the like and moving the location with a gesture. Yes.

(Third embodiment)
FIG. 13 is a flowchart employed in the third embodiment of the present invention. The flow of FIG. 13 shows processing that continues when the determination result of step S41 of FIG. 11 is NO. The hardware configuration in the present embodiment is the same as that in the second embodiment, and the operation flow other than that in FIG. 13 is the same as that in FIG.

  In each of the above-described embodiments, the shadow change image is created by obtaining the positional relationship between the light source and the target face. In the present embodiment, the information obtained in the signal processing and control unit 53 is recorded at the time of recording, so that it is possible to display a shadow change image effective at the time of subsequent photographing.

  FIG. 14 is an explanatory diagram for explaining tag information recorded at the same time when a photographed image is recorded. FIG. 14A shows a recorded captured image 90, and FIG. 14B shows tag information recorded simultaneously with respect to FIG. 14A. As shown in FIG. 14A, the recorded captured image 90 includes an image of a person 71 as a target object and an image of a lamp 72 as a light source.

  As shown in FIG. 14B, subject information, location and time information can be recorded as tag information while the captured image of FIG. 14A is being captured. Further, by executing each process of FIG. 11, as the tag information, the shadow change image generated at the time of shooting the image of FIG. 14A and the distance distribution information obtained for creating the shadow change image, It is possible to record information on the illumination direction (light source direction viewed from the object) φ.

  If the determination in step S41 in FIG. 11 is NO, the signal processing and control unit 53 proceeds to step S51 in FIG. 13 and has already captured a captured image of the same subject as the subject in the current captured image. It is determined whether it has been recorded. Furthermore, the signal processing and control unit 53 determines whether or not a shadow change image relating to an illumination direction different from the illumination direction of the current captured image is recorded from information on the illumination direction φ for the same subject. The signal processing and control unit 53 returns the processing to step S21 in FIG. 11 when there is no such recording, and proceeds to the next step S52 when there is recording. In step S52, the signal processing and control unit 53 reads out the searched shadow change image for reference display (step S53).

  FIG. 15 is an explanatory diagram showing a display example on the display screen 56b in this case. A through image 95 is displayed on the display screen 56b. In the through image 95, an image of the same person 96 as the person 71 of FIG. 14A and an image of the lamp 97 are displayed, and further, a shadow change image 98 obtained when the illumination direction from the lamp 97 is φ1 and A shadow change image 99 obtained when the illumination direction from the lamp 97 is φ2 is displayed. In addition, the shadow change images 98 and 99 also indicate that the illumination directions are φ1 and φ2, respectively.

  These shadow change images 98 and 99 are obtained when the captured image of FIG. 14A is captured. When the same person 96 shown in FIG. 15 is captured, these shadow change images 98 and 99 are displayed. By displaying it as a reference image in which the position of the illumination is changed, the photographer can obtain an extremely effective shooting suggestion.

  As described above, in the present embodiment, it is possible to store various information and shadow change images acquired at the time of shooting and display them as reference images at the time of current shooting, and the photographer can obtain a very effective suggestion regarding lighting. it can.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. A camera for moving images may be used, and a camera built in a personal digital assistant (PDA) such as a mobile phone or a smartphone may of course be used. Also, industrial and medical optical instruments such as endoscopes and microscopes may be used. Even if it is not a display device, it is possible to provide an observation device and a display device capable of displaying for capturing an image desired by the user by such a device. Needless to say, the object can be observed and diagnosed in detail by the position simulation of the light source and the object to be observed without photographing. In addition, it is not a technology that can be applied only to devices for obtaining captured images, but it can be used for advice by changing the position of objects and light sources on the screen based on images sent to remote locations. Therefore, the imaging unit is not an essential component.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  It should be noted that even if the operation flow in the claims, the description, and the drawings is described using “first,” “next,” etc. for convenience, it is essential to carry out in this order. It doesn't mean. In addition, it goes without saying that the steps constituting these operation flows can be omitted as appropriate for portions that do not affect the essence of the invention.

  Of the technologies described here, many of the controls and functions mainly described in the flowcharts can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. it can. As a computer program product, the program may be recorded or stored in whole or in part on a portable medium such as a non-volatile memory such as a flexible disk or a CD-ROM, or a storage medium such as a hard disk or a volatile memory. It can be distributed or provided at the time of product shipment or via a portable medium or communication line. The user can easily realize the display device of the present embodiment by downloading the program via a communication network and installing the program on the computer, or installing the program from the recording medium on the computer.

    DESCRIPTION OF SYMBOLS 10 ... Imaging part, 11 ... Shooting support part, 11a ... Face determination part, 11b ... Light source position determination part, 11c ... Shadow change part, 11d ... Display control part, 50 ... Camera, 52 ... Imaging part, 53 ... Signal processing And a control unit, 53a ... display control unit, 53b ... face detection unit, 53c ... shadow change unit, 53d ... distance distribution determination unit, 53e ... light source position determination unit, 57 ... recording unit.

Claims (7)

A position determination unit that determines the position of the object in the image and the position of the light source that illuminates the object;
Based on the positional relationship between the object and the light source, a shadow change unit that generates a shadow change image in which the shadow of the image of the object is corrected according to the distance from the light source;
A display device comprising: a display control unit configured to synthesize and display the shadow change image on the captured image. The shadow change unit, when displaying the shadow change image between the object and the light source in the captured image, the shadow having at least one of amplitude and brightness larger than the image of the object. When generating a modified image and displaying the shaded modified image in a direction opposite to the light source direction from the object in the captured image, at least one of amplitude and brightness is smaller than the image of the object. The display device according to claim 1, wherein a shadow change image is generated. The display control unit determines a display position of the shadow change image on a display screen on which a composite image of the captured image and the shadow change image is displayed;
The display device according to claim 1, wherein the shadow changing unit changes a shadow state of the shadow change image based on the change of the display position. A recording control unit for recording the positional relationship between the object and the light source and the shading-change image on a recording medium;
The display control unit reads the shade-changed image from the recording medium when the positional determination unit records information on the positional relationship different from the positional relationship between the object and the light source on the recording medium. The display device according to any one of claims 1 to 3, wherein the display device is combined with the captured image and displayed. A position determination step in which a captured image of a subject is given from the imaging unit, and the position of the object in the captured image and the position of the light source that illuminates the object are determined;
Based on the positional relationship between the object and the light source, a shadow change step for generating a shadow change image in which the shadow of the image of the object is corrected according to the distance from the light source;
A display control step comprising: a display control step of combining and displaying the shadow change image on the captured image. Receiving an operation of changing the positional relationship on the screen between the light source displayed in the screen and the object illuminated by the light source;
Forming a sub-image during the position change operation;
Changing the shadow obtained on the object according to the positional relationship and displaying it on the sub-image. On the computer,
A position determination step in which a captured image of a subject is given from the imaging unit, and the position of the object in the captured image and the position of the light source that illuminates the object are determined;
Based on the positional relationship between the object and the light source, a shadow change step for generating a shadow change image in which the shadow of the image of the object is corrected according to the distance from the light source;
A display control step for executing a display control step of combining and displaying the shadow-changed image on the captured image.