WO2019207922A1

WO2019207922A1 - Display imaging device

Info

Publication number: WO2019207922A1
Application number: PCT/JP2019/006080
Authority: WO
Inventors: 真治木村
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2018-04-26
Filing date: 2019-02-19
Publication date: 2019-10-31
Also published as: JP7002644B2; JPWO2019207922A1

Abstract

This display imaging device 7 is provided with: a transmission type display 1 which comprises a periodic structure member 15; a camera 2 which images the front side of the display 1 from the rear side of the display 1; and a generation unit 62 which generates a front image on the basis of a captured image obtained from the camera 2. The generation unit 62 generates a front image such that the periodic structure member 15 is not displayed on the basis of a plurality of captured images which are obtained when a light-receiving position R of the camera is at a plurality of locations and thus the periodic structure member 15 in the images is displayed at different locations.

Description

Display imaging device

The present invention relates to a display imaging device.

A method for realizing a video call with a line-of-sight match has been proposed. Japanese Patent Application Laid-Open No. 2004-133867 discloses a technique of capturing an image of a user in front of a display with a camera provided behind a transmissive display that displays a call partner. According to this method, since the front of the user can be imaged in a state where the user and the other party are facing each other, there is an increased possibility that a video call with a line of sight can be realized.

JP 2010-232828 A

In the transmissive display, members including signal lines and the like that exist for driving the display are periodically arranged. When such a member (hereinafter also referred to as “periodic structure member” or the like) is reflected in the camera, the quality (image quality) of the captured image is degraded. This problem is not studied at all in Patent Document 1. An object of one embodiment of the present invention is to provide a display and imaging device capable of realizing a state in which line-of-sight coincides and improving image quality.

A display and imaging apparatus according to an aspect of the present invention includes a transmissive display including a periodic structure member, a camera that images the front of the display from behind the display, and a generation unit that generates a front image based on the captured image of the camera And the generation unit acquires a period based on a plurality of captured images that are obtained when the light receiving position of the camera is at each of the plurality of positions and the periodic structure members in the image are displayed at different positions. A front image is generated so that the structural member is not displayed.

According to the above display and imaging device, the front of the display can be imaged from the front from the front of the display with the camera. When used for a video call or the like, the user in front of the display can be imaged from the front, so that a line-of-sight state can be realized. Further, based on a plurality of captured images displayed at different positions of the periodic structure member in the image, a front image is generated so that the periodic structure member is not displayed. Since the periodic structure member is not displayed in the front image, the image quality can be improved.

According to one aspect of the present invention, there is provided a display and imaging device capable of realizing a state of line-of-sight matching and improving image quality.

It is a schematic block diagram of the display imaging device which concerns on embodiment. It is a figure which shows notionally the front enlarged view of a part of display, and the sight of a camera. It is a figure which shows the block diagram of a control apparatus. It is a figure which shows the example of the method of acquiring the captured image in a some light reception position. It is a figure which shows the example of the production | generation of a front image. It is a flowchart which shows the example of the process performed by the control apparatus. It is a schematic block diagram of a video call system. It is a figure which shows the block diagram of a control apparatus. It is a figure which shows the example of the production | generation of a front image. It is a flowchart which shows the example of the process performed by the control apparatus. It is a figure which shows the example of hardware constitutions, such as a control apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

FIG. 1 shows a schematic configuration of a display imaging device 7 according to the embodiment. The display imaging device 7 includes a display 1, a camera 2, and a control device 6. The display 1, the camera 2, and the control device 6 are supported at a desired position by being supported by a support member (not shown), for example. FIG. 1 also shows a user U1 who is a user of the display imaging device 7. In some figures, an XYZ orthogonal coordinate system is shown. The X axis indicates the horizontal direction of the display 1. The Y axis indicates the vertical direction of the display 1. The Z axis indicates the front-rear direction of the display 1.

The display 1 is a transmissive display, for example, a self-luminous display in which each display pixel includes an organic light emitting diode (OLED: Organic Light Emitting Diode). A front surface 1 a of the display 1 is a display surface of the display 1. The display 1 has a viewing angle forward, and light (video light or the like) from the display 1 travels forward from the front surface 1a within the viewing angle range. The image displayed on the front surface 1a of the display 1 is brightest when the display 1 is viewed from the front.

The camera 2 is provided behind the display 1. The camera 2 images the front of the display 1 through the display 1. An example of the imaging target is the user U1. When the camera 2 repeats imaging at a predetermined imaging frame rate, a plurality of captured images that are continuous in time series are acquired. The plurality of captured images acquired in this way can constitute a video in front of the display 1.

The light from the imaging object located within the range of the angle of view of the camera 2 enters the camera 2. In FIG. 1, the spread of the angle of view of the camera 2 is conceptually shown using an angle α. The camera 2 receives incident light at the light receiving position R. The light receiving position R is the position of the imaging start point of the camera 2, and a scene viewed forward from the light receiving position R is a captured image of the camera 2. The camera 2 includes a photoelectric conversion element, an image processing circuit, and the like. The position of the photoelectric conversion element may be the light receiving position R of the camera 2.

The distance between the light receiving position R of the camera 2 and the back surface 1b of the display 1 is referred to as a distance A1 and is illustrated. A distance between the light receiving position R and the user U1 is referred to as a distance A2 and illustrated. The distance A1 is sufficiently smaller than the distance A2. The distance A2 is usually about several tens of centimeters to several meters, although it depends on the usage mode of the display imaging device 7. The distance A1 may be set to about several tenths to several hundredths of the distance A2, for example, about several mm to several cm. The camera 2 may be provided behind the display 1 in a state where the end 2a of the camera 2 is close to or in contact with the back surface 1b of the display 1 so that the distance A1 is as small as possible.

When the front side of the display 1 is imaged by the camera 2, the problem described with reference to FIG. FIG. 2A conceptually shows an enlarged front view of a part of the display 1. The display 1 includes a plurality of display pixels 11 and a periodic structure member 15. Each display pixel 11 may be a sub-pixel of the display 1, and in this case, each display pixel 11 includes a light emitting element corresponding to one of the three primary colors of R, G, and B. The example of a light emitting element is the above-mentioned organic light emitting diode. The periodic structure member 15 includes signal lines that exist in a mesh shape for driving the display 1. For example, the luminance value of the light emitting element of each display pixel 11 is adjusted in accordance with an electrical signal (control signal or the like) from the signal line. In addition, the periodic structure member 15 may include members necessary for signal line wiring. The periodic structure member 15 is also a lattice frame that partitions the display pixels 11 into a mesh shape. In the example shown in FIG. 2, one display pixel 11 is included in one mesh. However, a plurality of display pixels 11 may be included in one mesh. For example, when one signal line running in the horizontal direction drives the light emitting elements of the display pixels 11 located on both sides of the signal line, the two display pixels 11 may be included in one mesh in the vertical direction. . When one signal line running in the vertical direction drives the light emitting elements of the display pixels 11 located on both sides of the signal line, two display pixels 11 may be included in one mesh in the horizontal direction. Since the number of display pixels 11 included in one mesh is flexible, the shape of the mesh is not limited to a square, and may be, for example, a rectangle.

The periodic structure member 15 includes a plurality of horizontal members 15x and a plurality of vertical members 15y. Each horizontal member 15 x extends in the horizontal direction of the display 1 and is arranged at equal intervals in the vertical direction of the display 1. Each vertical member 15 y extends in the vertical direction of the display 1 and is arranged at equal intervals in the horizontal direction of the display 1. Details of the dimensions of the periodic structure member 15 will be described with reference to FIG.

A camera 2 that captures an image of the front of the display 1 via the display 1 shows a scene as shown in FIG. As illustrated, the periodic structure member 15 of the display 1 is reflected in the camera 2. The periodic structure member 15 usually appears as a grid frame of black or a color close to black. In FIG. 2B, the following symbols are attached regarding the dimensions of the periodic structure member 15.
Vertical member width Wx: The length of the vertical member 15y in the horizontal direction and the member width of the periodic structure member 15 in the horizontal direction.
Vertical member interval Dx: a horizontal distance between adjacent vertical members 15y. It is also the periodic interval (mesh size) of the periodic structural member 15 in the horizontal direction. The vertical member interval Dx may be larger than the vertical member width Wx (for example, twice or more).
Horizontal member width Wy: The length in the vertical direction of the horizontal member 15x, and the member width of the periodic structure member 15 in the vertical direction.
Horizontal member interval Dy: a vertical distance between adjacent horizontal members 15x. It is also the periodic interval (mesh size) of the periodic structure member 15 in the vertical direction. The horizontal member interval Dy may be larger than the horizontal member width Wy (for example, twice or more).

The size of the imaging pixel (not shown) of the camera 2 will be described. When compared with the mesh of the periodic structure member 15, the imaging pixels of the camera 2 may be sufficiently smaller than the mesh. If the number of horizontal imaging pixels in one mesh is Nx and the number of vertical imaging pixels is Ny, Nx and Ny may be several to several hundreds. The vertical member interval Dx and the vertical member interval Dx can also be expressed in units of the number of imaging pixels of the camera 2. In that case, the vertical member interval Dx is Nx, and the horizontal member interval Dy is Ny.

The dimensions related to the periodic structure member 15 described above, that is, the vertical member interval Dx, the vertical member width Wx, the horizontal member interval Dy, and the horizontal member width Wy may be measured, for example, by performing calibration in advance.

When compared with the line width of the periodic structure member 15, the size of the imaging pixel of the camera 2 may be equal to or less than the vertical member width Wx in the horizontal direction and less than or equal to the horizontal member width Wy in the vertical direction. That is, the vertical member width Wx and the horizontal member width Wy can be equal to or larger than the size of one imaging pixel of the camera 2. This is because, as described above, the distance A1 between the light receiving position R of the camera 2 and the back surface 1b of the display 1 is small, and the camera 2 images the periodic structure member 15 from a close range. In this case, the horizontal member 15x and the vertical member 15y in the captured image are displayed in the captured image, and the possibility that the periodic structure member 15 is reflected in the captured image increases. Note that only the horizontal member 15x may be reflected due to the difference in size between the vertical member width Wx and the horizontal member width Wy, or only the vertical member 15y may be reflected. Due to the reflection of the periodic structure member 15, the possibility that the quality (image quality) of the image captured by the camera 2 is lowered is also increased. This problem is improved by the control device 6 described below.

FIG. 3 shows a block diagram of the control device 6. The control device 6 is configured to be communicable with the camera 2, for example, and can perform various controls relating to imaging by the camera 2 and acquire a captured image of the camera 2. The control device 6 includes an acquisition unit 61 and a generation unit 62 as its functional blocks. The acquisition unit 61 can include a physical element (for example, a holding unit 61a described later) that moves the camera 2.

The acquisition unit 61 is a part that acquires a captured image of the camera 2 when the light receiving position R of the camera 2 is at each of a plurality of positions. The captured images acquired here are a plurality of captured images displayed at different positions of the periodic structure member 15. In one embodiment, the camera 2 is configured to be movable. The light receiving position R of the camera 2 moves between a plurality of positions in accordance with the movement of the camera 2. For example, in the method illustrated in FIG. 4A, the acquisition unit 61 functions as a moving unit that moves the camera 2 itself by controlling the holding unit 61 a that holds the camera 2. As will be described later, the camera 2 may be controlled to vibrate (micro). The holding | maintenance part 61a is realizable in various aspects using a well-known method. The holding unit 61a may be, for example, a component that moves microscopically using a vibrator. Instead of moving the holding unit 61a, the holding unit 61a may be attached to the main body of the camera 2, an image sensor (CMOS, CCD, etc.), a lens, or the like. It may be realized by providing a vibrator and giving a minute movement, or the camera 2 is slightly vibrated by controlling the light reflection direction with a MEMS (Micro Electro Mechanical Systems) mirror provided inside the camera 2. You may implement | achieve by control which image | photographs the same image | video as a state.

In one embodiment, the light receiving positions R of the camera 2 exist simultaneously at a plurality of positions. As such a camera, for example, a light field camera, a 3D camera, or the like can be used. FIG. 4B shows a schematic configuration of the optical system of the light field camera 21. An imaging target of the light field camera 21 is conceptually illustrated as an imaging target 30. In the light field camera 21, a microlens array 23 is disposed in front of the image sensor 22 between the main lens 24 and an image sensor (sensor) 22 such as a CMOS / CCD. Since the principle of the light field camera is well known, detailed description is omitted. For example, if the light field camera 21 is used, a plurality of captured images (image groups) similar to those when the light receiving position R of the camera 2 is moved can be acquired without moving the camera itself. Moreover, such an image group can be acquired by one imaging (one shot). In this case, a value obtained by dividing the number of captured images constituting the image group by the time required for one shot can be regarded as the captured frame rate. According to the light field camera 21, the movement speed of the light receiving position R of the camera 2 is substantially reduced since an operation system for causing movement like the holding unit 61a shown in FIG. In the case of vibration, the same effect as increasing the vibration frequency) is obtained. Thereby, the imaging frame rate of the camera 2 can be increased.

Thereafter, as described with reference to FIG. 4A, the light receiving position R of the camera 2 moves between a plurality of positions, and the captured image of the camera 2 at each position is mainly acquired. explain. On the other hand, as described with reference to FIG. 4B, in the form in which the light receiving positions R of the camera 2 are simultaneously present at a plurality of positions, it corresponds to each position to which the light receiving position R of the camera 2 is moved. A light field camera or the like adjusted to have a plurality of light receiving positions may be used. In that case, what is necessary is just to understand that the movement of the light reception position R of the camera 2 in the following description is unnecessary.

3, the acquisition unit 61 moves the light receiving position R of the camera 2 to a plurality of positions. In one embodiment, the acquisition unit 61 may move the light receiving position R of the camera 2 in a direction inclined with respect to the horizontal direction and the vertical direction (hereinafter referred to as “inclination direction”). The plurality of positions may be defined by coordinates (XY plane coordinates) parallel to the back surface 1b of the display 1. The plurality of positions may be the same position in the Z-axis direction. In this case, even if the light receiving position R of the camera 2 moves, the distance A1 (see FIG. 1) between the light receiving position R of the camera 2 and the back surface 1b does not change.

The plurality of positions are determined so that minute parallax (fine parallax) is generated between the captured images acquired at the respective light receiving positions. The occurrence of fine parallax means that in the captured images acquired at the respective light receiving positions, the imaging target is displayed at approximately the same position (overlapping), while the periodic structure member 15 is displayed at a different position (not overlapping). ) Means that parallax occurs between captured images. Such fine parallax occurs because the distance A1 between the light receiving position R of the camera 2 and the back surface 1b of the display 1 depends on the light receiving position R and the imaging target, as described above with reference to FIG. This is because it is smaller than the distance A2 between the user U1. The plurality of positions are arranged side by side in a straight line in the movement direction of the plurality of positions (for example, the above-described tilt direction). The plurality of positions may be three or more positions.

A distance between the light receiving positions R of the camera 2 for causing fine parallax (movement distance of the light receiving position R) will be described. First, since the distance A1 between the light receiving position R of the camera 2 and the back surface 1b of the display 1 is sufficiently small, the movement of the light receiving position R of the camera 2 on the XY plane includes the periodic structure member 15 of the display 1. It can be regarded as movement on a plane. Therefore, the periodic structure member 15 in the captured image of the camera 2 moves as much as the moving distance of the light receiving position R of the camera 2. Here, the minimum unit of the moving distance of the light receiving position R of the camera 2 is defined as follows.
Movement distance Δx: Distance in the horizontal direction (separation distance) between adjacent positions in the movement direction (for example, the tilt direction). Depending on how the coordinates are set, Δx may be a negative value.
Movement distance Δy: Distance in the vertical direction (separation distance) between adjacent positions in the movement direction. Depending on how the coordinates are set, Δy may be a negative value.
As described above, the moving distance Δx and the moving distance Δy can take negative values, but hereinafter, the magnitude of the moving distance Δx and the magnitude of the moving distance Δy are simply referred to as the moving distance Δx and the moving distance Δy.

The moving distance Δx and the moving distance Δy are set as follows using the dimensions of the periodic structure member 15 shown in FIG. The movement distance Δx is set to be not less than the vertical member width Wx and less than the vertical member interval Dx. That is, Wx ≦ | Δx | <Dx. The movement distance Δy is set to be not less than the horizontal member width Wy and less than the horizontal member interval Dy. That is, Wy ≦ | Δy | <Dy. In the control of the acquisition unit 61 described later, the movement distance Δx is set to be less than half of the vertical member interval Dx, that is, Wx ≦ | Δx | <(Dx) / 2. The movement distance Δy is set to be less than half of the horizontal member interval Dy, that is, Wy ≦ | Δy | <(Dy) / 2.

As described above, the vertical member interval Dx and the vertical member interval Dx can be represented by the number of pixels Nx and Ny of the camera 2. In that case, the movement distance Δx is expressed as less than Nx or less than (Nx / 2). The movement distance Δy is expressed as less than Ny or less than (Ny / 2).

The technical significance of the movement distance Δx and the movement distance Δy will be described. Since the movement distance Δx is equal to or greater than the vertical member width Wx, the vertical member 15y does not overlap at the same position in the captured image acquired before and after the movement of the light receiving position R of the camera 2. Further, since the movement distance Δx is less than the vertical member interval Dx, different vertical members 15y (for example, adjacent vertical members 15y) overlap at the same position in the captured image obtained before and after the movement of the light receiving position R of the camera 2. There is nothing. Furthermore, when the movement distance Δx is less than half of the vertical member interval Dx, the vertical member 15y does not overlap even at the same position in the three captured images acquired over the two movements of the light receiving position R.

Similarly, since the movement distance Δy is equal to or greater than the horizontal member width Wy, the horizontal member 15x does not overlap at the same position in the captured image acquired before and after the movement. Further, since the movement distance Δy is less than the horizontal member interval Dy, different horizontal members 15x (for example, adjacent horizontal members 15x) do not overlap at the same position in the images acquired before and after the movement. Further, when the movement distance Δy is less than half of the horizontal member interval Dy, the horizontal member 15x does not overlap even at the same position in the three captured images acquired over the two movements of the light receiving position R.

If the horizontal member width Wy is so small that the horizontal member 15x does not appear in the captured image of the camera 2, the moving distance Δx can be made sufficiently small. In that case, the movement distance Δx may be zero. When the vertical member width Wx is so small that the vertical member 15y is not reflected in the captured image of the camera 2, the movement distance Δy can also be made sufficiently small. In that case, the movement distance Δy may be zero.

When the acquisition unit 61 moves the light receiving position R of the camera 2 in the inclination direction, the angle of the inclination direction with respect to the horizontal direction (inclination angle) is calculated by using the movement distance Δx and the movement distance Δy, and the inclination angle = arctan (Δy / Δx ). The tilt angle is less than 90 °. If the movement distance Δx and the movement distance Δy are the same, the angle in the tilt direction is set to 45 °. Considering that the moving distance Δx and the moving distance Δy may vary depending on the vertical member width Wx, the vertical member interval Dx, the horizontal member width Wy, and the vertical member interval Dx, the inclination angle may be a value different from 45 °. Good. When the movement distance Δx and the movement distance Δy are close to some extent, for example, the inclination angle can be a size within a range of 45 ° ± 10 °, 45 ° ± 20 °, or 45 ° ± 30 °.

When one of the movement distances Δx and Δy is zero, the inclination angle is 0 ° or 90 °. In that case, the acquisition unit 61 moves the light receiving position R of the camera 2 substantially in the horizontal direction or the vertical direction.

3, the acquisition unit 61 moves the light receiving position R of the camera 2 in conjunction with the imaging frame rate of the camera 2. Specifically, when the captured image is acquired (at the time of imaging), the acquisition unit 61 moves the light receiving position R of the camera 2 so that the light receiving position R of the camera 2 is located at one of a plurality of positions. When the imaging frame rate of the camera 2 is 60 fps, the acquisition unit 61 moves the light receiving position R of the camera 2 60 times per second.

Furthermore, the acquisition unit 61 may reciprocate the light receiving position R of the camera 2. In this case, the acquisition unit 61 may vibrate the light receiving position R of the camera 2 (at a high speed and minutely) as described above with reference to FIG. The length of one of the reciprocating paths (that is, the width of vibration) may be set to be less than the vertical member interval Dx in the horizontal direction and less than the horizontal member interval Dy in the vertical direction.

Returning to FIG. 3, the generation unit 62 will be described. The generation unit 62 is a part that generates an image in front of the display 1 (hereinafter also referred to as “front image”) based on the captured image acquired by the acquisition unit 61. The generation unit 62 generates a front image so that the periodic structure member 15 is not displayed. Such a front image is an image in which the reflection of the periodic structure member 15 is reduced as compared with a captured image of the camera 2 (for example, an image as shown in FIG. 2B). The front image may be an image without the periodic structure member 15 (removed).

When the acquisition unit 61 reciprocates the light receiving position R of the camera 2, the generation unit 62 moves forward based on the captured images of the camera 2 that are sequentially acquired when the light receiving position R of the camera 2 is in each position of the reciprocating path. An image may be generated. A time series median filter may be used to generate the forward image by the generation unit 62. Since the median filter itself is a known technique, the description thereof is omitted here. The function of the median filter may be provided in the generation unit 62 itself, or the function of the median filter provided in other elements (may be a server (not shown) outside the control device 6) is generated. The unit 62 may use it.

When using the median filter, the generation unit 62 applies the median filter to captured images acquired at least three light receiving positions arranged in the moving direction (for example, the tilt direction) of the light receiving position R among the plurality of positions. The resulting image may be generated as a front image. For example, an image obtained by combining image pickup pixels whose luminance value is the median luminance value of each of the three image pickup pixels is generated as a front image. The captured images acquired when the light receiving position R of the camera 2 is at each of the three positions are the three images in time series acquired when the light receiving position R of the camera 2 moves in conjunction with the imaging frame. It is an image. A median filter is applied to a plurality of such time-series captured images (image group). Hereinafter, a case where there are three positions will be described.

An example of generation of a front image by the generation unit 62 will be described with reference to FIG. 5A to 5C show captured images acquired when the light receiving position R of the camera 2 is at each of the three positions. In the figure, an arrow AR indicates the moving direction of the light receiving position R. Since the light receiving position R reciprocates as described above, the arrow AR indicates the direction of the forward or backward path of the reciprocating path. The light receiving positions R of the camera 2 from which the captured images shown in FIGS. 5A to 5C are acquired are referred to as light receiving positions R0 to R2. Here, one coordinate of the lattice point of the periodic structure member 15 in the captured image is represented as P (x, y). The lattice point is a point where the horizontal member 15 x and the vertical member 15 y intersect in the periodic structure member 15.

FIG. 5A shows a captured image acquired at the light receiving position R0. In the captured image, the periodic structure member 15 is present in front of the user U1 who is the imaging target. One coordinate of the lattice point of the periodic structure member 15 is P (x0, y0).

FIG. 5B shows a captured image acquired at the light receiving position R1. The light receiving position R1 is a position moved from the light receiving position R0 by the moving distance Δx and the moving distance Δy. In this captured image, the coordinates of the lattice point are P (x1, y1), and x1 = x0 + Δx and y1 = y0 + Δy. Even in the captured image acquired at the light receiving position R1, the periodic structure member 15 is present in front of the user U1. In the captured image acquired at the light receiving position R1, fine parallax occurs with respect to the captured image acquired at the light receiving position R. Therefore, the position of the user U1 is the same while the position of the periodic structure member 15 is moved. Yes.

(C) in FIG. 5 shows a captured image acquired at the light receiving position R2. The light receiving position R2 is a position moved from the light receiving position R1 by the moving distance Δx and the moving distance Δy. In this captured image, the coordinates of the lattice point are P (x2, y2), and x2 = x1 + Δx and y1 = y1 + Δy. Even in the captured image acquired at the light receiving position R2, the periodic structure member 15 is present in front of the user U1. In the captured image acquired at the light receiving position R2, fine parallax occurs with respect to the captured image acquired at the light receiving position R and the light receiving position R1, so that the position of the user U1 is the same while the position of the periodic structure member 15 is Is moving.

Here, as described above, the movement distance Δx and the movement distance Δy are set to Wx ≦ | Δx | <(Dx) / 2 and Wy ≦ | Δy | <(Dy) / 2, respectively. In this case, the vertical member 15y does not overlap and the horizontal member 15x does not overlap at the same position in the three captured images acquired at the respective light receiving positions R0 to R2. Therefore, when imaging pixels at the same position in the three imaging images are compared, at most one imaging pixel displays the vertical member 15y among the three imaging pixels. At least two imaging pixels are imaging pixels that display an imaging target (for example, user U1) instead of the vertical member 15y. Similarly, among the three imaging pixels, at most one imaging pixel displays the horizontal member 15x. At least two imaging pixels are imaging pixels that display an imaging target, not the horizontal member 15x. Therefore, if a median filter is applied to these three captured images, the horizontal member 15x and the vertical member 15y of the periodic structure member 15 as shown in FIG. 5D are not displayed (the reflection is reduced). I) One captured image is obtained, and this captured image can be generated as a forward image.

Here, when the acquisition unit 61 reciprocates the light receiving position R of the camera 2, after the light receiving position R moves from the light receiving position R0 to the light receiving position R2 along the direction of the arrow AR, this time, the arrow AR It moves from the light receiving position R2 to the light receiving position R0 along the opposite direction. When the arrow AR is the forward direction of the reciprocating path, the light receiving position R passes through the light receiving position R0, the light receiving position R1, and the light receiving position R2 in this order in the forward path, and the light receiving position R is the light receiving position R2, the light receiving position R1, and the light receiving position in the return path Go through R0 in this order. The generation unit 62 generates a first front image based on the captured image acquired when the light receiving position R of the camera 2 is in each position of the forward path of the reciprocating path (light receiving position R0 to light receiving position R2). A second front image may be generated based on a captured image acquired when the vehicle is at each position of the return path (light receiving position R2 to light receiving position R0) of the path. These first front image and second front image may be repeatedly generated while the reciprocal movement continues. Here, the light receiving position R2 is a return position from the forward path to the return path, and the light reception position R0 is a return position from the return path to the return path. The captured image acquired when the light receiving position R of the camera 2 is at the turn-back position of these round-trip paths may be used in common for generating the first front image and the second front image. Specifically, the captured image acquired at the light receiving position R0 and the light receiving position R2 is one of the three captured images acquired on the forward path of the reciprocating path (the light receiving position R0 to the light receiving position R2). It is also one captured image of the three captured images acquired on the return path (light reception position R2 to light reception position R0) of the round-trip path. Therefore, the captured image acquired at the light receiving position R2 includes a median filter that is applied to the three captured images acquired in the forward path, and a median that is applied to the three captured images acquired in the subsequent return path. It may be used in common with the filter. The captured image acquired at the light receiving position R0 includes a median filter that is applied to the three captured images acquired in the return path and a median that is applied to the three captured images acquired in the subsequent outbound path. It may be used in common with the filter.

The generation timing of the front image by the generation unit 62 will be described. First, the imaging time t of the camera 2 is expressed as t = t0, t1, t2, t3, t4, t5, t6, t7, t8, t9. If the time t interval is Δt, Δt is the reciprocal of the imaging frame rate of the camera 2. Next, the time t ′ to which the forward image generated by the generation unit 62 corresponds is expressed as t ′ = t′0, t′1, t′2, t′3. In this case, the forward image corresponding to time t′0 is generated based on three captured images (time-series image group) acquired at times t0 to t2 (each light receiving position R). The forward image corresponding to time t′1 is generated based on the three captured images acquired at times t2 to t4. The forward image corresponding to time t′2 is generated based on the three captured images acquired at times t4 to t6. The forward image corresponding to time t′3 is generated based on the three captured images acquired at times t6 to t8. The same applies to time t ′ thereafter. In this case, two intervals of time t, which are times t = t0, t2, t4, t6, t8... Correspond to one interval of time t ′. Therefore, if the time t ′ interval is Δt ′, Δt ′ is twice Δt. When the imaging frame rate of the camera 2 is 60 fps, Δt is 1/60 second and Δt ′ is 1/30 second. That is, the video display frame rate when the front image generated by the generation unit 62 is displayed is 30 fps, which is a half of the imaging frame rate 60 fps of the camera 2.

The time t ′ may be associated with any one of the three times t, and may be the same time as the last time t of the three times t, for example. In this case, time t′0 is time t2. Time t′1 is time t4. Time t′2 is time t6. Time t′3 is time t8. The same applies to time t ′ thereafter.

FIG. 6 is a flowchart showing an example of processing (image generation method) executed by the control device 6. The acquisition unit 61 reciprocates the light receiving position R of the camera 2 as described above.

In step S1, the generation unit 62 obtains the luminance value f (t) of each imaging pixel in the captured image acquired at the light receiving position R at time t. The light receiving position R here is, for example, the light receiving position R0 in FIG. 5A in the case of the reciprocating movement, and the light receiving position R2 in FIG. 5C in the case of the return path in the reciprocating movement. is there. The luminance value f (t) is represented by a numerical value, and the numerical value may be larger as the luminance is higher.

In step S2, the acquisition unit 61 moves the light receiving position R of the camera 2 diagonally (in the tilt direction) by a minute amount. This is because the acquisition unit 61 reciprocates the light receiving position R of the camera 2. The minute amounts are the movement distance Δx and the movement distance Δy. Since the time required for the movement is Δt, the time when the movement is completed is t + Δt. In the captured image acquired after the movement, fine parallax occurs with respect to the captured image acquired before the movement.

In step S3, the generation unit 62 obtains the luminance value f (t + Δt) of each imaging pixel in the captured image acquired at the light receiving position R at time t + Δt. The light receiving position R here is, for example, the light receiving position R1 of FIG.

In step S4, the acquisition unit 61 moves the light receiving position R of the camera 2 diagonally by a minute amount. That is, the light receiving position R of the camera 2 further moves in the tilt direction by the movement distance Δx and the movement distance Δy. The time at which this movement is completed is t + 2Δt. In the captured image acquired after the movement, fine parallax occurs with respect to the captured image acquired before the movement.

In step S5, the generation unit 62 obtains the luminance value f (t + 2Δt) of each imaging pixel of the captured image acquired at the light receiving position at time t + 2Δt. The light receiving position R here is, for example, the light receiving position R2 in FIG. 5C in the case of the forward path of the reciprocating path, and the light receiving position R0 in FIG. 5A in the case of the return path of the reciprocating path. is there.

In step S6, the generation unit 62 applies a median filter with the luminance value f (t), the luminance value f (t + Δt), and the luminance value f (t + 2Δt).

In step S7, the generation unit 62 sets the luminance value after the median filter at time t ′ to f (t ′). The luminance value f (t ′) may be, for example, the luminance value of the median value of the three luminance values of the luminance value f (t), the luminance value f (t + Δt), and the luminance value f (t + 2Δt). An image obtained by applying the median filter is generated as a forward image at time t ′. The time t ′ may be the same time as the time t + 2Δt. The forward image generated in this way is a forward image in which the periodic structure member 15 is not displayed as described above with reference to FIG.

In step S8, the acquisition unit 61 switches the moving direction of the light receiving position R of the camera 2. That is, the moving direction of the light receiving position R is switched between the forward path and the return path of the reciprocating path. Thereafter, the process returns to step S2. The reason why the process returns to step S2 instead of step S1 is that, as described above, the captured image acquired at the return position of the round-trip path is used in common on the forward path and the return path. Therefore, the generation unit 62 uses the luminance value f (t + 2Δt) obtained in the previous step S5 as the luminance value f (t) of each imaging pixel in the captured image at time t. That is, after the captured image information at time t + 2Δt used in the previous loop is substituted for the captured image information at time t used in the next loop, the processes in steps S2 to S8 are repeatedly executed.

By repeatedly executing the processing of the flowchart as described above, a forward image is repeatedly generated so that the periodic structure member 15 as described above is not displayed. When the loop of the flowchart corresponds to the forward path of the round trip path of the light receiving position R, the front image generated in step S7 is the first front image described above. When the loop of the flowchart corresponds to the return path of the round trip path of the light receiving position R, the front image generated in step S7 is the second front image described above. By repeatedly executing the processing of the flowchart, the first front image and the second front image are repeatedly generated.

In the above-described steps S2 and S4, the example in which the light receiving position R of the camera 2 is moved obliquely by a small amount has been described. However, as described above, when the movement distance Δx or the movement distance Δy is either zero, The light receiving position R is moved not horizontally but horizontally or vertically. Further, as described above with reference to FIG. 4B, in the form in which the light receiving positions R of the camera 2 are simultaneously present at a plurality of positions, the above-described steps S2 and S4 are not necessary. The process of “switching the moving direction of the light receiving position” in step S8 is also unnecessary.

The display imaging device 7 described above includes a display 1, a camera 2, and a control device 6. The display 1 includes a periodic structure member 15. The camera 2 images the front of the display 1 from behind the display 1. The control device 6 includes a generation unit 62. The generation unit 62 generates a front image of the display 1 based on the captured image of the camera 2. More specifically, the generation unit 62 acquires the plurality of captured images in which the periodic structure member 15 is displayed at different positions by being acquired when the light receiving position R of the camera 2 is at each of the plurality of positions. Based on this, a forward image is generated so that the periodic structure member 15 is not displayed.

First, according to the display imaging device 7, the front of the display 1 can be imaged from the front by the camera 2 from behind the display 1. When used for a video call or the like, the user U1 in front of the display 1 can be imaged from the front, so that a line-of-sight state can be realized.

An example of a video call will be described with reference to FIG. A video call system 9 shown in FIG. 7 includes a control unit 5 and a communication device 8 in addition to the display imaging device 7. Although not shown in FIG. 7, as described above, the display imaging device 7 includes the camera 2 and the control device 6 (see FIG. 1). In this example, the user U1 of the video call system 9 is making a video call with the user U2 who is a call partner. The user U1 is viewing the display 1 from the front (Z-axis positive direction). Therefore, the camera 2 can image the user U1 from the front (Z-axis negative direction) via the display 1 (FIG. 2 and the like). The communication device 8 transmits and receives video data, audio data, and the like to and from a video call system (not shown) used by the user U2. The video data includes video data of user U1 and video data of user U2. The audio data includes user U1 audio data and user U2 audio data.

The control unit 5 is a part that executes various controls necessary for operating the display imaging device 7 in the video call system 9. The control unit 5 receives display data from the display 1 (for example, video data of the user U2) by the communication device 8, and transmits captured image data from the camera 2 (for example, video data of the user U1) by the communication device 8. The transmission / reception process is executed. The control unit 5 may be a constituent element of the display imaging device 7. In addition, elements necessary for the video call system 9 such as a speaker and a microphone (not shown) may be included in the display imaging device 7.

Furthermore, according to the display imaging device 7, as described so far, a front image is generated such that the periodic structure member 15 is not displayed. Therefore, the image quality can be improved. When used in the video call system 9, the video of the user U1 is presented to the user U2, but the video quality can be improved because the periodic structure member 15 is not reflected in the video of the user U1. .

The light receiving position R of the camera 2 may be moved between a plurality of positions by moving the camera 2 as described above with reference to FIG. Thereby, it is possible to acquire a captured image when the light receiving position R of the camera 2 is at each of a plurality of positions using a normal camera that is not a light field camera, a 3D camera, or the like.

As described above with reference to FIG. 4B, a light field camera, a 3D camera, or the like in which the light receiving positions R exist simultaneously at a plurality of positions may be used. In this case, for example, an effect similar to that of increasing the moving speed of the light receiving position R of the camera 2 can be obtained because the operation system for moving the camera 2 becomes unnecessary. Thereby, the imaging frame rate of the camera 2 can be increased.

When the acquisition unit 61 reciprocates the light reception position R of the camera 2, the generation unit 62 captures the images of the camera 2 that are sequentially acquired when the light reception position R of the camera 2 is at each position on a reciprocation path passing through a plurality of positions. A forward image may be generated based on the image. In this case, the plurality of positions may include a return position of the round-trip path. The generation unit 62 generates a first captured image based on the captured images of the camera 2 acquired when the light receiving positions R of the camera 2 are at a plurality of positions on the forward path of the round-trip path, and generates a plurality of images on the return path of the round-trip path. A second front image may be generated based on the captured image of the camera 2 acquired when the camera is at the position. The generation unit 62 may generate the first front image and the second front image by using the captured image of the camera 2 acquired at the turn-back position of the round-trip path in common. For example, when the captured image of the camera 2 at the return position of the round-trip path is not used in common, one front image is generated every time three captured images are acquired, so the video display frame rate of the front image Decreases to a frame rate that is one third of the imaging frame rate of the camera 2. On the other hand, if the captured image of the camera 2 at the turn-back position of the round-trip path is used in common, one front image is generated every time two captured images are acquired, so the video display frame rate of the front image is set. , It can be increased to half of the imaging frame rate of the camera 2.

The periodic structure member 15 may include a horizontal member 15x and a vertical member 15y. The horizontal members 15 x may extend in the horizontal direction of the display 1 and be arranged at equal intervals in the vertical direction of the display 1. The vertical members 15 y may extend in the vertical direction of the display 1 and be arranged at equal intervals in the horizontal direction of the display 1. The plurality of positions may be arranged in an inclined direction with respect to the horizontal direction and the vertical direction. When the display 1 includes such a periodic structure member 15, if the movement of the light receiving position R of the camera 2 is only in the horizontal direction, the position of the vertical member in each captured image can be prevented from overlapping, but the position of the horizontal member is Duplicate. If the movement of the light receiving position R of the camera 2 is only in the vertical direction, the position of the horizontal member in each captured image can be prevented from overlapping, but the position of the vertical member overlaps. By moving the light receiving position R of the camera 2 in the tilt direction, the position of the horizontal member and the position of the vertical member in each captured image can be prevented from overlapping. Therefore, the reflection of the periodic structure member 15 in the front image generated by the generation unit 62 can be further reduced, and the image quality can be further improved.

The distance between adjacent positions among the plurality of positions is equal to or greater than the member width (vertical member width Wx, horizontal member width Wy) of the periodic structure member 15 and the periodic interval (vertical member interval Dx, horizontal member interval Dy). It may be less than half. The generation unit 62 displays an image obtained by applying a median filter to the captured image of the camera 2 acquired when the light receiving position R of the camera 2 is at each of three positions. It may be generated as an image.

If the distance between a plurality of positions is set as described above, when imaging pixels at the same position in the three captured images are compared, there are many imaging pixels that display the vertical member 15y among the three imaging pixels. Both are one. At least two imaging pixels are imaging pixels that display an imaging target (for example, user U1) instead of the vertical member 15y. Similarly, among the three imaging pixels, at most one imaging pixel displays the horizontal member 15x. At least two imaging pixels are imaging pixels that display an imaging target, not the horizontal member 15x. Therefore, if a median filter is applied to these three captured images, a captured image in which the periodic structure member 15 is not displayed (such that the periodic structure member 15 is removed) is obtained, and this captured image is displayed on the display 1. Can be generated as a forward image.

[Modification]
In the above-described embodiment, an example has been described in which a front image is generated such that the periodic structure member 15 is not displayed based on captured images acquired at at least three light receiving positions. On the other hand, according to the modification described below, it is possible to generate a front image in which the periodic structure member 15 is not displayed based on captured images acquired at at least two light receiving positions.

FIG. 8 shows a block diagram of a control device 6A included in the display imaging device 7A according to such a modification. The control device 6A is different from the control device 6 (FIG. 3) in that it includes an acquisition unit 61A and a generation unit 62A instead of the acquisition unit 61 and the generation unit 62, and further includes a storage unit 63.

The acquisition unit 61A reciprocates the light receiving position R of the camera 2 in the same manner as the acquisition unit 61 (FIG. 3). In the case of a light field camera or the like, the movement is not necessary as described above. Here, the acquisition unit 61 reciprocates the light receiving position R of the camera 2 so as to pass through at least three positions, whereas the acquisition unit 61A reciprocates the light reception position R of the camera 2 so as to pass through at least two positions. Move. The two positions may be the return positions of the round trip path. Hereinafter, a case where the acquisition unit 61A reciprocates the light receiving position R of the camera 2 so as to pass through two positions will be described.

The movement of the light receiving position R of the camera 2 by the acquisition unit 61A is also determined so that the fine parallax described so far occurs. The minimum unit of the moving distance of the light receiving position R of the camera 2 by the acquisition unit 61A is defined as follows.
Movement distance ΔxA: a horizontal distance (separation distance) between two positions. Depending on how the coordinates are set, ΔxA may be a negative value.
Movement distance ΔyA: a vertical distance (separation distance) between two positions. Depending on how the coordinates are set, ΔyA may be a negative value.
Hereinafter, the magnitude of the movement distance ΔxA is simply referred to as a movement distance ΔxA. The magnitude of the movement distance ΔyA is simply referred to as the movement distance ΔyA.

The moving distance ΔxA is set to be not less than the vertical member width Wx and less than the vertical member interval Dx. That is, Wx ≦ | ΔxA | <Dx. The movement distance ΔyA is set to be not less than the horizontal member width Wy and less than the horizontal member interval Dy. That is, Wy ≦ | ΔyA | <Dy.

The generating unit 62A is a part that generates a front image based on a captured image acquired when the light receiving position R of the camera 2 is at each of the two positions. Specifically, the generation unit 62A does not use the imaging pixel that displays the periodic structure member 15 among the imaging pixels at the same position in the two captured images so that the periodic structure member 15 is not displayed (or removed). A forward image is generated.

The storage unit 63 is a part that stores in advance information related to the imaging pixels that display the periodic structure member 15. As an example of such information, the storage unit 63 stores in advance the luminance value L of the imaging pixel when the imaging pixel of the captured image of the camera 2 displays the periodic structure member 15. The luminance value L may be a value measured in advance by calibration. Even if the measurement is not performed, a sufficiently small value (for example, the minimum value) may be set as the luminance value L using the low luminance value of the imaging pixel at the position of the periodic structure member 15.

The generation unit 62A generates a front image using an imaging pixel having a luminance value that is distant from the luminance value L stored in the storage unit 63 among the imaging pixels of the two captured images. This is because an imaging pixel whose luminance value is close to the luminance value L is likely to be an imaging pixel that displays the periodic structure member 15, so that such an imaging pixel is not used. For example, when the luminance value of one imaging pixel is 10 and the luminance value of the other imaging pixel is 40 out of the imaging pixels at the same position in two captured images, it is separated from the luminance value L (in this case, less than 10). An imaging pixel having a luminance value 40 of the existing one is employed.

An example of generation of a front image by the generation unit 62A will be described with reference to FIG. FIG. 9A shows a captured image acquired at the light receiving position R0A. One coordinate of the lattice point of the periodic structure member 15 is PA (x0A, y0A).

FIG. 9B shows a captured image acquired at the light receiving position R1A. The light receiving position R1A is a position moved from the light receiving position R0A by the moving distance ΔxA and the moving distance ΔyA. In this captured image, the coordinates of the lattice points are PA (x1A, y1A), and x1A = x0A + ΔxA and y1A = y0A + ΔyA. In the captured image acquired at the light receiving position R1A, since the fine parallax is generated with respect to the captured image acquired at the light receiving position R0A, the position of the periodic structure member 15 is moved while the position of the user U1 is the same. is doing.

Here, as described above, the movement distance ΔxA and the movement distance ΔyA are set to Wx ≦ | ΔxA | <Dx and Wy ≦ | ΔyA | <Dy, respectively. In this case, the vertical member 15y does not overlap and the horizontal member 15x does not overlap at the same position in the two captured images at the light receiving position R0A and the light receiving position R1A. Therefore, when the imaging pixels at the same position in the two captured images are compared, the luminance value of the imaging pixel displaying the horizontal member 15x and / or the vertical member 15y out of the two imaging pixels is the horizontal member 15x and / or The brightness value is closer to the brightness value L stored in the storage unit 63 than the brightness value of the imaging pixels not displaying the vertical member 15y. Therefore, the periodic structure member 15 as shown in (c) of FIG. 9 is removed by combining the imaging pixels having the luminance values far from the luminance value L out of the imaging pixels of these two captured images. Only one captured image is obtained, and this captured image can be generated as a front image in which the periodic structure member 15 is not displayed.

As described above, the acquisition unit 61A moves the light receiving position R of the camera 2 back and forth. Therefore, the generation unit 62A generates a first front image based on the captured images acquired at the light receiving position R0A and the light receiving position R1 in the forward path of the round trip path, and at the light receiving position R1A and the light receiving position R0A in the return path of the round trip path. A second front image may be generated based on the acquired captured image. For the generation of the first front image and the second front image, a captured image acquired at the return position of the round-trip path can be used in common. In this case, the video frame rate when displaying the forward image generated by the generation unit 62A is the same as the imaging frame rate of the camera 2. For example, when the imaging frame rate of the camera 2 is 60 fps, the video frame rate of the front image is also 60 fps.

FIG. 10 is a flowchart showing an example of processing (image generation method) executed by the control device 6A.

In step S11, the generation unit 62A obtains the luminance value f (t) of each imaging pixel in the captured image at the light receiving position R at time t. The light receiving position R here is, for example, the light receiving position R0A in FIG. 9A in the case of the forward path of the round-trip path, and is the light receiving position R1A in FIG. 9B in the case of the return path in the round-trip path. is there.

In step S12, the acquisition unit 61A moves the light receiving position R of the camera 2 diagonally by a minute amount. The minute amount is the above-described movement distance ΔxA and movement distance ΔyA. Since the time required for the movement is Δt, the time when the movement is completed is t + Δt. In the captured image acquired after the movement, a fine parallax occurs with respect to the imaging pixel acquired before the movement.

In step S13, the generation unit 62A obtains the luminance value f (t + Δt) of each imaging pixel in the captured image acquired at the light receiving position R at time t + Δt. The light receiving position R here is, for example, the light receiving position R1A of FIG. 9B in the case of the forward path of the reciprocating path, and the light receiving position R0A of FIG. 9A in the case of the return path of the reciprocating path. is there.

In step S14, the generation unit 62A uses the luminance value L stored in the storage unit 63, and the difference value (| f (t) between the luminance value f (t) obtained in the previous step S11 and the luminance value L. ) −L |) and a difference value (| f (t + Δt) −L |) between the luminance value f (t + Δt) obtained in the previous step S13 and the luminance value L.

In step S15, the generation unit 62A sets the luminance value used for calculation of the difference value having the larger value among the two difference values obtained in the previous step S14 as f (t ′). That is, an image obtained by combining imaging pixels having a luminance value far from the luminance value L out of the two luminance values obtained in the previous step S11 and step S13 is a front corresponding to the time t ′. Generated as an image. The time t ′ may be the same time as the time t + Δt in the previous step S13.

In step S16, the acquisition unit 61A switches the moving direction of the light receiving position R of the camera 2. That is, the moving direction of the light receiving position R is switched between the forward path and the return path of the reciprocating path. Thereafter, the process returns to step S12 again. The reason why the process returns to step S12 instead of step S11 is that, as described above, the captured image acquired at the return position of the round-trip path is used in common on the forward path and the return path. Therefore, the generation unit 62A uses the luminance value f (t + Δt) obtained in the previous step S13 as the luminance value f (t) of each imaging pixel in the captured image at time t. That is, the information of the captured image at time t + Δt used in the previous loop is substituted into the information of the captured image at time t used in the next loop, and then the processing of steps S12 to S16 is repeatedly executed.

By repeatedly executing the processing of the flowchart as described above, a forward image is repeatedly generated so that the periodic structure member 15 as described above is not displayed. When the loop in the flowchart corresponds to the forward path of the round trip path of the light receiving position R, the front image generated in step S16 is the first front image described above. When the loop of the flowchart corresponds to the return path of the round trip path of the light receiving position R, the front image generated in step S16 is the second front image described above. By repeatedly executing the processing of the flowchart, the first front image and the second front image are repeatedly generated.

In the above-described step S12, the example in which the light receiving position R of the camera 2 is moved by a minute amount is described. However, as described above, if the moving distance Δx or the moving distance Δy is zero, the light receiving position R is received. The position R is moved not horizontally but horizontally or vertically. Further, as described above with reference to FIG. 4B, in the form in which the light receiving positions R of the camera 2 are simultaneously present at a plurality of positions, the process of step S12 described above is not necessary. The process of “switching the moving direction of the light receiving position” in step S16 described above is also unnecessary.

In the display imaging device 7A according to the modified example described above, the storage unit 63 stores in advance the luminance value of the imaging pixel when the imaging pixel of the captured image of the camera 2 displays the periodic structure member 15. The distance between adjacent positions among the plurality of positions is not less than the member width (vertical member width Wx, horizontal member width Wy) of the periodic structure member 15 and less than the periodic interval (vertical member interval Dx, horizontal member interval Dy). It is. The generation unit 62 is stored in the storage unit 63 out of each imaging pixel in the captured image of the camera 2 acquired when the light receiving position R of the camera 2 is at each of two positions among the plurality of positions. An image obtained by combining image pickup pixels having a luminance value away from the luminance value L is generated as a front image in which the periodic structure member 15 is not displayed (steps S14 and S15).

When imaging pixels at the same position in two captured images are compared, the luminance value of the imaging pixel where the horizontal member 15x and / or the vertical member 15y is located among the two imaging pixels is the horizontal member 15x and / or the vertical member. It is closer to the luminance value L stored in the storage unit 63 than the luminance value of the imaging pixel in which 15y is not located. Therefore, by combining the imaging pixels having the luminance values that are separated from the luminance value L among the imaging pixels of these two captured images, one captured image in which the periodic structure member 15 is not displayed is obtained. The captured image can be generated as a front image of the display 1. In this case, since the front image can be generated using only two captured images, the video frame rate of the front image can be increased as compared with the case where the front image is generated using three captured images.

As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. In the above embodiment, the example in which the display imaging device 7 is used in the video call system 9 has been described. However, the use of the display imaging device 7 is not limited to the video call system 9. For example, the display imaging device 7 may be used for the user U1 to display his / her own video. In this case, the display 1 displays the video imaged by the camera 2. Accordingly, the user U1 can take a picture while confirming how the user U1 is appearing without having to worry about the position of the camera 2 simply by facing the display 1. Such an application of the display imaging device 7 has an advantage that, in a photographer (certification photographer, sticker printer) or the like, the photographing preview screen can be confirmed with the eye line facing forward at the time of photographing. Even when augmented reality (AR) signage placed in a shopping center or the like is used to synthesize CG with a user's video, it can be enjoyed more naturally. In addition, applications that deliver live images while reflecting themselves will be able to deliver with a more natural look.

Note that the block diagram used in the description of the above embodiment shows functional unit blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one device physically and / or logically coupled, and two or more devices physically and / or logically separated may be directly and / or indirectly. (For example, wired and / or wireless) and may be realized by these plural devices.

For example, the control device 6 or the like according to an embodiment of the present invention may function as a computer that performs processing such as the acquisition unit 61 and the generation unit 62. FIG. 11 is a diagram illustrating an example of a hardware configuration of the control device 6 and the like according to the present embodiment. Hereinafter, the control device 6 will be described as an example. The control unit 5, the control device 6 </ b> A, and the communication device 8 can be described in the same manner as the control device 6. The control device 6 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the term “apparatus” can be read as a circuit, a device, a unit, or the like. The hardware configuration of the control device 6 may be configured to include one or a plurality of the devices illustrated in the figure, or may be configured not to include some devices.

Each function in the control device 6 is read by a predetermined software (program) on hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs an operation and performs communication by the communication device 1004, in the memory 1002 and the storage 1003. This is realized by controlling reading and / or writing of data.

The processor 1001 controls the entire computer by operating an operating system, for example. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, a register, and the like. For example, the

acquisition units

61 and 61A and the

generation units

62 and 62A described above may be realized by the processor 1001.

Further, the processor 1001 reads a program (program code), a software module, and data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above embodiments is used. For example, the acquisition unit 61 and the generation unit 62 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and may be realized similarly for other functional blocks. Although the above-described various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via a telecommunication line.

The memory 1002 is a computer-readable recording medium, and includes, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (ElectricallyrErasable Programmable ROM), RAM (Random Access Memory), and the like. May be. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (program code), a software module, and the like that can be executed to implement the wireless communication method according to the embodiment of the present invention.

The storage 1003 is a computer-readable recording medium such as an optical disc such as a CD-ROM (Compact Disc ROM), a hard disc drive, a flexible disc, a magneto-optical disc (eg, a compact disc, a digital versatile disc, a Blu-ray). (Registered trademark) disk, smart card, flash memory (for example, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, or the like. The storage 1003 may be referred to as an auxiliary storage device. The storage medium described above may be, for example, a database, server, or other suitable medium including the memory 1002 and / or the storage 1003.

The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also called a network device, a network controller, a network card, a communication module, or the like. For example, the communication device 8 described above may be realized by the communication device 1004.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts an external input. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Also, the devices such as the processor 1001 and the memory 1002 are connected by a bus 1007 for communicating information. The bus 1007 may be configured with a single bus or may be configured with different buses between apparatuses.

The control device 6 includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). A part or all of each functional block may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these hardware.

As mentioned above, although this embodiment was described in detail, it is clear for those skilled in the art that this embodiment is not limited to embodiment described in this specification. The present embodiment can be implemented as a modification and change without departing from the spirit and scope of the present invention defined by the description of the scope of claims. Therefore, the description of the present specification is for illustrative purposes and does not have any limiting meaning to the present embodiment.

The processing procedures, sequences, flowcharts, and the like of each aspect / embodiment described in this specification may be switched in order as long as there is no contradiction. For example, the methods described herein present the elements of the various steps in an exemplary order and are not limited to the specific order presented.

As used herein, the phrase “based on” does not mean “based only on”, unless expressly specified otherwise. In other words, the phrase “based on” means both “based only on” and “based at least on.”

In the present specification, when a designation such as “first”, “second”, etc. is used, any reference to that element does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to the first and second elements does not mean that only two elements can be employed there, or that in some way the first element must precede the second element.

These terms are similar to the term “comprising” as long as “include”, “including” and variations thereof are used herein or in the claims. It is intended to be comprehensive. Furthermore, the term “or” as used herein or in the claims is not intended to be an exclusive OR.

In this specification, unless there is only one device that is clearly present in context or technically, a plurality of devices are also included.

In the whole of the present disclosure, a plural is included unless it is clearly indicated by a context.

DESCRIPTION OF SYMBOLS 1 ... Display, 2 ... Camera, 7, 7A ... Display imaging device, 15 ... Periodic structure member, 15x ... Horizontal member, 15y ... Vertical member, 61, 61A ... Acquisition part, 62, 62A ... Generation part, Dx ... Vertical member Spacing, Dy ... horizontal member spacing, Wx ... vertical member width, Wy ... horizontal member width.

Claims

A transmissive display including a periodic structure member;
A camera that images the front of the display from behind the display;
A generating unit that generates a front image based on a captured image of the camera;
With
The periodic structure member is acquired based on a plurality of captured images in which the periodic structure member in the image is displayed at different positions by being acquired when the light receiving position of the camera is at each of a plurality of positions. Generate a forward image so that is not displayed,
Comprising
Display imaging device.
The light receiving position of the camera moves between the plurality of positions according to the movement of the camera.
The display imaging device according to claim 1.
The light receiving position of the camera is simultaneously present at the plurality of positions.
The display imaging device according to claim 1.
The generation unit generates the front image based on captured images of the camera sequentially acquired when a light receiving position of the camera is at each position of a round-trip path passing through the plurality of positions.
The plurality of positions includes a return position of the round-trip path,
The generation unit generates a first front image based on captured images of the camera acquired when a light receiving position of the camera is at a plurality of positions in the forward path of the round-trip path, and in a return path of the round-trip path Generating a second front image based on the captured image of the camera acquired when the image is at a plurality of positions;
The generation unit generates the first front image and the second front image by using in common the captured image of the camera acquired when the light receiving position of the camera is at the return position of the round-trip path. To
The display imaging apparatus according to any one of claims 1 to 3.
The periodic structure member is:
A plurality of horizontal members extending in the horizontal direction of the display and arranged at equal intervals in the vertical direction of the display;
A plurality of vertical members extending in the vertical direction and arranged at equal intervals in the horizontal direction;
Including
The plurality of positions are arranged in an inclined direction with respect to the horizontal direction and the vertical direction.
The display imaging device according to any one of claims 1 to 4.
The distance between adjacent positions among the plurality of positions is equal to or greater than the member width of the periodic structure member and less than half of the periodic interval.
The generation unit obtains an image obtained by applying a median filter to a captured image of the camera acquired when the light receiving position of the camera is at each of three positions of the plurality of positions. Generating as the front image,
The display imaging device according to any one of claims 1 to 5.
When the imaging pixel of the captured image of the camera displays the periodic structure member, the storage unit further stores a luminance value of the imaging pixel in advance,
The distance between adjacent positions among the plurality of positions is equal to or greater than the member width of the periodic structure member and less than the periodic interval.
The generation unit stores, in the storage unit, among each imaging pixel in the captured image of the camera acquired when the light receiving position of the camera is at each of two positions of the plurality of positions. An image obtained by combining imaging pixels having a luminance value that is distant from the luminance value being generated is generated as the front image;
The display imaging device according to any one of claims 1 to 5.