JP2019132916A - Signage device - Google Patents

Abstract

To acquire information of an observer highly precisely.SOLUTION: The present invention includes: an optical element 2 for forming a spacial image by forming an image in the air out of light from an image display device 1, which shows an image display 11; a beam splitter 4 for transmitting a part of the light from the image display device partially and reflecting the other light; a camera 5 arranged so that an axis 5a of the light through the beam splitter is perpendicular to a two-dimensional spacial image 12; and an acquisition unit for acquiring state information of a person in an image taken by the camera; and a state information storage unit for storing the state information of the person acquired by the acquisition unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a signage device using aerial video display.

In recent years, an aerial image display device has been proposed in which a real image of an object is formed in the air using a special optical plate so that an observer can observe the real image.
For example, Patent Document 1 uses an optical plate having a plurality of reflecting surfaces (corner mirror arrays) orthogonal to each other in plan view, and forms an aerial image at a plane-symmetric position with the object and the optical plate in between. Technology is disclosed.
If such an aerial image is realized on a large screen, the eye catching property is high, and use in the field of digital signage is being studied.

On the other hand, in the field of digital signage, not only information is displayed, but a camera is arranged to photograph an observer who is observing the screen, and information on the observer is obtained from the photographed image. The idea of switching to the corresponding screen display has been proposed.
For example, the signage device of Patent Document 2 determines whether there is a person in front of the digital signage device based on the image captured by the camera, whether the person is smiling based on the person's face recognition, and different audio depending on the determination. The content is being output.
In addition, the signage device of Patent Document 3 captures the surrounding state with a camera, and changes the display contents depending on whether a person is in front of the display.

Japanese Patent No. 4865088 JP 2017-59916 A JP 2017-116693 A

However, since all of the signage devices of Patent Documents 2 and 3 have to place a camera avoiding a display device that displays information, the observer who is observing the display device is positioned away from the front. I could only shoot from
In the case of such a photographed image, it can be performed without any problem as long as the presence / absence of a person is detected, but there is a risk that sufficient accuracy cannot be obtained for more detailed detection such as face recognition and facial expression recognition. was there.

  An object of the present invention is to provide a signage device that can obtain information of an observer with higher accuracy by photographing.

The invention according to claim 1 is a signage device,
An optical element that forms an aerial image by focusing light from an image display device that displays a display image in the air; and
A beam splitter that transmits part of the light from the image display device and reflects the other part;
A camera arranged such that an optical axis passing through the beam splitter is perpendicular to the planar aerial image;
An acquisition unit that acquires state information of a person included in an image captured by the camera;
And a state information storage unit that stores the state information of the person acquired by the acquisition unit.

The invention according to claim 2 is the signage device according to claim 1,
It has a light emission part which irradiates invisible light toward the regular observation position with respect to the aerial image.

The invention according to claim 3 is the signage device according to claim 2,
The irradiation range of the invisible light by the light emitting unit is adjusted to the size of the shooting range of the camera at a predetermined observation position for the aerial image.

The invention according to claim 4 is the signage device according to claim 2 or 3,
The camera performs imaging through a filter member that cuts visible light.

The invention according to claim 5 is the signage device according to any one of claims 2 to 4,
The beam splitter has a semi-transmissive layer that transmits visible light and reflects invisible light on a surface thereof.

The invention according to claim 6 is the signage device according to any one of claims 1 to 5,
The camera is a stereo camera.

The invention according to claim 7 is the signage device according to any one of claims 1 to 6,
A person detection unit for detecting a person included in the shooting range of the camera from the shot image of the camera;
A first display control unit that displays the aerial image when the person is detected by the person detection unit, and terminates the display of the aerial image after a certain period of time when the person is no longer detected by the person detection unit; It is characterized by providing.

The invention according to claim 8 is the signage device according to any one of claims 1 to 7,
An observation determination unit for determining whether or not the aerial image is observed from the orientation of a person's face included in the photographed image of the camera;
The acquisition unit is configured to measure the observation time as the state information of the person when the observation determination unit determines that the aerial image is being observed.

The invention according to claim 9 is the signage device according to any one of claims 1 to 8,
A second display control unit that sequentially displays a plurality of display images on the image display device;
The acquisition unit detects a facial expression of a person included in an image captured by the camera, and when the detected facial expression is a specified facial expression, the display image displayed at that time is one of the plurality of display images. And the facial expression of the person is stored in the state information storage unit as the state information of the person.

The invention according to claim 10 is the signage device according to any one of claims 1 to 9,
A second display control unit that sequentially displays a plurality of display images on the image display device;
The acquisition unit is characterized by acquiring the state information of the person classified for each of the plurality of display images.

The invention according to claim 11 is the signage device according to any one of claims 1 to 10,
The optical element is an optical plate having a plurality of reflecting surfaces orthogonal to each other when viewed from a direction perpendicular to the plate plane.

The invention according to claim 12 is the signage device according to any one of claims 1 to 10,
The optical element is a retroreflector having a retroreflective surface that reflects along the incident direction of incident light.

  According to said structure, it becomes possible to provide the signage apparatus which can obtain an observer's information more accurately.

It is explanatory drawing which shows typically the structure of the whole signage apparatus which is 1st embodiment. It is a perspective view which shows schematic structure of an optical plate. It is a perspective view of the single plate-shaped member which comprises the said optical plate. It is a perspective view of the single other plate-shaped member which comprises the said optical plate. It is explanatory drawing which shows the imaging principle of the real image in two dimensions. It is explanatory drawing which shows typically reflection of the light ray in three-dimensional space. It is explanatory drawing which shows typically a mode that the several light ray emitted from 1 point condenses to 1 point via a separate reflective surface in 3D space. It is explanatory drawing which shows the relationship between the imaging | photography range of a camera, and the irradiation range of a light emission part. It is a block diagram which shows the control system of the signage apparatus of FIG. It is a flowchart which shows control of the signage apparatus which a control apparatus performs. It is explanatory drawing which shows the dimension of each part in the case where it shoots with the optical axis of a camera perpendicularly | vertically with respect to an imaging surface, and the optical axis of a camera inclines with respect to the normal line of an imaging surface. It is explanatory drawing which shows the example which added the total reflection mirror 52 to the signage apparatus of FIG. It is explanatory drawing which shows typically the whole structure of the signage apparatus which is 2nd embodiment. It is explanatory drawing which showed the principle in which the light emitted from one point by a retroreflector forms an image in one point in the air. It is explanatory drawing which shows the other example of arrangement | positioning of the camera and retroreflection body inside a signage apparatus. It is explanatory drawing which shows the example of the signage apparatus carrying a stereo camera.

[First embodiment]
A signage apparatus 10 according to a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is an explanatory diagram schematically showing the overall configuration of the signage device 10. The signage device 10 is an image display device 1 that displays a display image 11 as a planar object, and an optical element that forms an aerial image 12 as a real image by forming light from the display image 11 in the air. The optical plate 2, a beam splitter 4 that transmits part of the light from the display image 11 and reflects the other part, and an optical axis 5 a that passes through the beam splitter 4 is perpendicular to the planar aerial image 12. A camera 5 arranged so as to cross the light, a light emitting unit 7 for projecting detection light to the front side of the planar aerial image 12, a housing 8 for storing and supporting these components, and control of each of the above components The control device 9 is mainly provided.

The signage device 10 guides light from the display image 11 of the image display device 1 to the air on the side opposite to the light incident side with respect to the planar optical plate 2, so that the optical plate 2 is plane-symmetric with respect to the plane. An aerial image 12 that is a real image of the display image 11 is formed at a position.
The aerial image 12 displays an image having the same content as the display image 11, and the signage device 10 captures a person observing the aerial image 12 with the camera 5. Furthermore, the control device 9 of the signage device 10 acquires state information of a person photographed based on the photographed image, and stores the state information in a state information storage unit 99 described later.

[Optical plate]
Next, details of the above-described optical plate 2 will be described. FIG. 2 is a perspective view showing a schematic configuration of the optical plate 2.
One direction parallel to the plate plane of the optical plate 2 is a Y direction, a direction parallel to the plate plane and perpendicular to the Y direction is an X direction, and a direction perpendicular to the plate plane is a Z direction. In FIG. 1, the Y direction and the Z direction are parallel to the paper surface, and the X direction is perpendicular to the paper surface.
The optical plate 2 has a two-layer structure in which two optical panels 20 and 30 are stacked in the Z direction.
The optical panel 20 is formed by laminating a plurality of long plate-like members 21 along the Y direction in the X direction. The optical panel 30 is formed by laminating and arranging a plurality of plate-like members 31 that are long along the X direction in the Y direction.

  FIG. 3 is a perspective view of a single plate member 21. The plate-like member 21 has a rectangular parallelepiped transparent substrate 211 made of transparent glass or resin. The transparent substrate 211 is long in the Y direction, and a reflective surface 212 is formed on one surface of two opposing planes along the YZ plane by vapor deposition of a metal such as aluminum. . In addition, the reflective surface 212 may be formed on both surfaces of two opposing planes along the YZ plane of the transparent substrate 211.

  FIG. 4 is a perspective view of a single plate member 31. The plate member 31 has a rectangular parallelepiped transparent substrate 311 made of transparent glass or resin. The transparent substrate 311 is long in the X direction, and a reflective surface 312 is formed on one surface of two opposing planes along the XZ plane by vapor deposition of a metal such as aluminum. . Note that the reflecting surfaces 312 may also be formed on both surfaces of two opposing planes along the XZ plane of the transparent substrate 311.

By laminating and arranging a plurality of plate-like members 21 extending in the Y direction in the X direction, a plurality of reflective surfaces 212 along the YZ plane are arranged at intervals corresponding to the width of the plate-like member 21 in the X direction. Arranged side by side. Similarly, by arranging a plurality of plate-like members 31 extending in the X direction in a stacked manner in the Y direction, the plurality of reflecting surfaces 312 along the XZ plane correspond to the width of the plate-like member 31 in the Y direction. Arranged in the Y direction at intervals. And by such arrangement | positioning of several plate-shaped members 21 and 31, the reflective surface 212 of each plate-shaped member 21 and the reflective surface 312 of each plate-shaped member 31 are mutually orthogonally-positioned seeing from a Z-axis direction. It becomes.
That is, the optical plate 2 has an optical element structure called a corner mirror array.

Next, the principle of image formation by the optical plate 2 will be described. FIG. 5 shows the imaging principle of a real image in two dimensions (in the XZ plane). A plurality of light rays emitted from the point light source P are respectively reflected by the reflecting surface 212 parallel to the Z axis, and the position P ₀ opposite to the point light source P with respect to the X axis (with respect to the point light source P and the X axis). The light is symmetric). Thereby, a real image of the point light source P is formed at the position P ₀ .
The same applies to the formation of a real image in the YZ plane, and a plurality of light beams emitted from the point light source P are respectively reflected by the reflecting surface 312 parallel to the Z axis, and are pointed with respect to the Y axis. The light is condensed at a position P ₀ opposite to the light source P (a position symmetrical to the point light source P and the X axis). Thereby, a real image of the point light source P is formed at the position P ₀ .

FIG. 6 schematically shows the reflection of light rays in a three-dimensional space (XYZ coordinate system). In the three-dimensional space, the light beam A emitted from the point light source O is decomposed into a light beam A1 in the XZ plane and a light beam A2 in the YZ plane. By considering the reflection of A2 in the XZ plane or the YZ plane, the intersection of the ray A with the Z axis can be obtained. That is, the light ray A1 in the XZ plane is reflected by the reflecting surface 212 parallel to the YZ plane and then goes to the Z axis, and the light ray A2 in the YZ plane is parallel to the XZ plane. After being reflected by the reflecting surface 312, it goes to the Z-axis. These light rays A1 and A2 intersect at one point on the Z axis, that is, at the point O ₀ . Therefore, the light beam A is reflected by the reflecting surface 212 and the reflecting surface 312 a total of two times, and then travels toward the point O ₀ on the Z axis.

FIG. 7 schematically shows a state in which a plurality of light beams emitted from the point light source O are condensed at one point via separate reflecting surfaces in a three-dimensional space. A plurality of light rays emitted from the point light source O are reflected by the reflecting surface 212 parallel to the YZ plane and the reflecting surface 312 parallel to the XZ plane in the same manner as in FIG. Concentrate on O ₀ . Thereby, a real image of the point light source O is formed at the point O ₀ .

  In the above description, the imaging principle of the optical plate 2 having the two-layer structure in which the optical panels 20 and 30 are stacked is shown. The imaging principle is the same in the optical plate 2 having a single layer structure in which a plurality of reflecting surfaces parallel to the XZ plane and a plurality of reflecting surfaces parallel to the YZ plane are actually orthogonal to each other. .

[Image display device]
The image display device 1 includes a display device that displays a flat display image 11 such as an LCD (Liquid Crystal Display), an EL (ElectroLuminescent) display, or a CRT (Cathode Ray Tube).
Note that the image display device 1 is not limited to the above configuration, and it is also possible to use an apparatus that projects the display image 11 such as a projector.

In the image display device 1, the display unit that displays the display image 11 has a planar shape, and the display unit faces one side (referred to as the incident surface 22) of the optical plate 2 arranged in parallel to the XY plane. Are inclined with respect to the XY plane by an angle θ1 about the X axis. This angle θ1 is a direction inclined 45 ° counterclockwise on the paper surface of FIG.
The inclination angle θ1 is 45 °, which is the most desirable angle for improving the image quality of the aerial image 12. However, if the image quality of the aerial image 12 is within the allowable range, the inclination angle θ1 has a certain angle range centered on 45 °. It can be increased or decreased.
The incident surface 22 of the optical plate 2 is large enough to project at least the entire display image 11 when the display image 11 is projected toward the incident surface 22 along the normal line of the display image 11. It is desirable to set this. However, when only the central portion of the display image 11 is an image display area that is substantially necessary, the size is set so that at least only such an image display area that is substantially necessary can be projected. May be.

  When the display device 11 displays the display image 11, the aerial image 12 is formed on the optical plate 2 at a position that is plane-symmetric with respect to the optical plate 2. That is, the aerial image 12 is inclined by an angle θ2 (= 45 °) clockwise with respect to the XY plane on the paper surface of FIG.

[Light emitting part]
The light emitting unit 7 includes a plurality of LEDs (light emitting diodes) as light sources that emit infrared light that is invisible light, and a light projecting optical system that projects the infrared light emitted from the light source within a predetermined range. ing.
The light emitting unit 7 irradiates the face of the observer with infrared light in order to photograph the face of the person in front of the signage device 10, particularly the person (referred to as the observer) who observes the aerial image 12 with the camera 5. . For this reason, as shown in FIG. 8, infrared light is irradiated toward the prescribed observation position K with respect to the aerial image 12 so that infrared rays can be effectively irradiated onto the face of the observer. The light projecting optical system of the light emitting unit 7 matches the irradiation range H of the infrared school with the size of the shooting range S of the camera 5 at the specified observation position K with respect to the aerial image 12.
The specified observation position K is a position that is a predetermined distance away from the aerial image 12 formed at a fixed image formation position along the optical axis 5a of the camera 5, that is, the viewer visually displays the display contents. The position is easy to recognize, and is set in advance according to the size of the aerial image 12.
Then, the imaging range S at the observation position K corresponding to the angle of view determined by the optical system of the camera 5 and the irradiation range H of the light emitting unit 7 at the observation position K are set to substantially coincide.

[Beam splitter]
As shown in FIG. 1, the beam splitter 4 has a flat plate shape parallel to the XY plane, and is disposed on the surface opposite to the incident surface 22 of the optical plate 2 (referred to as an output surface 23). Yes.
The beam splitter 4 uses a so-called half mirror in which both the light transmittance and the reflectance are 50%. However, the beam splitter 4 is not limited to this, and the beam splitter 4 has a transmittance greater than the reflectance or the transmittance is the reflectance. Smaller ones can also be used.

By providing this beam splitter 4, the light from the optical plate 2 can be transmitted when the aerial image 12 is formed, and the light from the aerial image 12 side can be reflected to the camera 5 side. .
Although FIG. 1 shows an example in which the beam splitter 4 is arranged so as to be in close contact with the optical plate 2, if the optical axis 5 a of the camera 5 can be directed perpendicular to the aerial image 12, the beam splitter 4. May be separated from the optical plate 2 or may be arranged in an inclined direction with respect to the XY plane.

  The beam splitter 4 is desirably set to a size that allows the entire display image 11 to pass through. However, in the case where only the portion excluding the outer edge portion of the display image 11 is an image display area that is substantially necessary, at least the size of such an image display area that is substantially necessary can be transmitted. You may set it.

Further, the beam splitter 4 is formed with a semi-transmissive layer 41 that transmits visible light and reflects infrared light as invisible light over the entire surface on the aerial image 12 side.
The semi-transmissive layer 41 is made of a film material in which a plurality of polyester films having a property of reflecting infrared light are laminated, and the semi-transmissive layer 41 is applied to the entire surface of the beam splitter 4 on the aerial image 12 side. It is worn.
As the film material, for example, “Scotch Tint (registered trademark) auto film crystallin” manufactured by 3M is suitable.

[camera]
The camera 5 includes an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and an optical system.
In the optical system of the camera 5, as shown in FIG. 1, the optical axis 5 a is reflected by the semi-transmissive layer 41 on the surface of the beam splitter 4 on the side of the aerial image 12 and enters perpendicularly to the aerial image 12. Are arranged as follows. That is, the optical axis 5a of the optical system of the camera 5 is directed in a direction inclined by an angle θ3 (= 45 °) counterclockwise around the X axis with respect to the normal line of the beam splitter 4 on the paper surface of FIG. ing.

Note that “vertical” when the optical axis 5a of the optical system of the camera 5 is perpendicularly incident on the aerial image 12 via the beam splitter 4 is preferably strictly vertical, but substantially vertical. Range (substantially vertical range). That is, when the optical axis 5a is tilted with respect to the vertical, the resolution of the picked-up image is lowered according to the tilt angle, as shown in Equation (5) that approximates the rate of decrease in resolution described later. An inclination within a sufficiently small range can be regarded as substantially “vertical”.
For example, an angular range within ± 10 ° with respect to the vertical is included in the “vertical” range because the approximate resolution reduction rate is suppressed to about 96%.

The camera 5 is equipped with a filter member 51 that cuts visible light, and photographing is performed through the filter member 51.
As described above, the camera 5 shoots a shooting range and a person in the shooting range by the reflection of the beam splitter 4. At this time, the surroundings and background of the person are included in the shooting range. As described above, the imaging range S at the observation position K corresponding to the angle of view determined by the optical system of the camera 5 and the irradiation of the light emitting unit 7 at the observation position K. Since the range H is set to substantially coincide with the range H, the reflection of the background where the infrared light does not reach is suppressed, and the face of the person irradiated with the infrared light can be photographed exclusively.

[Control device]
FIG. 9 is a block diagram showing a control system of the signage device 10.
The control device 9 is connected to the image display device 1, the light emitting unit 7, and the camera 5 and comprehensively controls them. The control device 9 is provided with an image display device 1, a light emitting unit 7, a drive circuit and interface for driving the camera 5, and an image processing device for processing captured image data of the camera 5. Omitted.

The control device 9 displays an image storage unit 93 including a plurality of types of display image data to be displayed as a display image 11 as content in the image display device 1, and causes the image display device 1 to display the display image 11 according to these display image data. First and second display control units 91 and 92 are provided.
Further, the control device 9 includes a photographing control unit 94 that continuously performs photographing with the camera 5 at a predetermined cycle.
Further, the control device 9 detects a person included in the shooting range of the camera 5 from the image captured by the camera 5, acquires a person detection unit 95 that detects the face of the person, and obtains status information of the person included in the captured image. An acquisition unit 98, an observation determination unit 96 that determines whether or not the aerial image 12 is observed from the detected face direction of the person, and a state information storage unit 99 that stores the acquired state information of the person It has.

The control device 9 includes a central processing unit (CPU), a random access memory (RAM), an auxiliary storage device, and the like connected via a bus. The auxiliary storage device is configured using a nonvolatile storage device. The nonvolatile storage device referred to here is a so-called ROM (Read-Only Memory: including EP (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), mask ROM, etc.), FRAM (registered trademark) (Ferroelectric RAM), flash memory, Includes hard disks.
In FIG. 9, various functions of the control device 9 are shown in blocks. That is, the control device 9 loads the various programs (OS, applications, etc.) stored in the auxiliary storage device into the RAM and executes them by the CPU, whereby the first and second display control units 91 and 92, person detection It functions as an apparatus including the unit 95, the observation determination unit 96, the acquisition unit 98, and the like. In addition, the 1st and 2nd display control parts 91 and 92, the person detection part 95, the observation determination part 96, and the acquisition part 98 are implement | achieved when a program is run by CPU. The first and second display control units 91 and 92, the person detection unit 95, the observation determination unit 96, and the acquisition unit 98 may be configured as dedicated chips.
Further, the image storage unit 93 and the state information storage unit 99 are realized by a storage area set in a part of the auxiliary storage device, but these may be configured as a dedicated storage chip.

  The first display control unit 91 displays the aerial image 12 when a person is detected in the shooting range by the person detection unit 95, and when a person is no longer detected in the shooting range by the person detection unit 95, after a certain period of time has elapsed. The image display device 1 is controlled to end the display of the aerial image 12. The elapsed time for ending the display is arbitrary, but it is desirable to select an appropriate value as appropriate.

The second display control unit 92 displays display images corresponding to a plurality of contents in a predetermined order from the start to the end of the display of the aerial image 12 by the first display control unit 91. 1 is controlled.
In the above-described imaging control unit 94, imaging by the camera 5 is executed in a short period regardless of whether the aerial image 12 is being displayed or not displayed by the first display control unit 91.

The person detection unit 95 detects a person in the shooting range based on the shot image and the face when there is a person.
The person detection unit 95 detects a person or a face in a photographed image by using a pattern recognition classifier whose parameters are determined in advance by machine learning using a large amount of sample data. This is done using these discriminators.

  In the determination of whether or not there is a person within the shooting range in the person detection unit 95, a difference image between the current shot image and the previous shot image is obtained, and the presence of the person is determined based on the presence or absence of a moving object. Also good. In this case, if a moving object other than a person, for example, a moving POP advertisement for advertising, is reflected in the shooting range, the determination accuracy is lowered. Therefore, it is desirable to add a determination to exclude moving objects that repeat regular operations.

The acquisition unit 98 detects state information such as the height, sex, age, face orientation, and facial expression of the photographed image using the person and face detected by the person detection unit 95.
In the case of the acquisition unit 98, pattern recognition classifiers for status information such as sex, age, face orientation, and facial expression are individually provided, and status information is detected using these classifiers.
For example, regarding a gender, a male discriminator is provided, and when it is identified as a male, it is determined as a male, and otherwise it is determined as a female (a configuration including a female discriminator may be used).
The age is determined by preparing a classifier for each band of a certain range of age, and the face direction is determined by preparing a classifier for each of multiple face directions. The determination is made by preparing discriminators for various facial expressions such as indifference.
The height is calculated from the ratio of the person detected by the person detection unit 95 to the face and the height of the person.

Then, the acquisition unit 98 stores state information such as the detected height, sex, age, face orientation, and facial expression of the detected person in the state information storage unit 99.
At this time, the acquisition unit 98 determines which of the display images corresponding to the plurality of contents the display image displayed by the second display control unit 92 when the captured image is captured is the second display control unit 92. And the displayed display image is associated with state information such as a person's height, gender, age, face orientation, facial expression, and the like, and recorded in the state information storage unit 99.
This makes it possible to conduct research such as what height, what gender, and what age a person looks at with any expression on the display image of each content.

The acquisition unit 98 detects the facial expression of the person included in the photographed image of the camera 5, and the displayed image displayed at that time is the displayed image regardless of the detected facial expression. Although it is acquired as state information of a person in association with the presence, and stored in the state information storage unit 99, the present invention is not limited to this.
For example, the display displayed at that time only when it is determined that the facial expression detected by the acquisition unit 98 is a specific facial expression, for example, a facial expression that attracts interest in the display image. A configuration may be adopted in which a facial expression is stored as person state information in association with which display image the image is.

The observation determination unit 96 determines whether or not the aerial image 12 is observed based on the face orientation of the person detected by the acquisition unit 98. Specifically, the observation determination unit 96 determines that the aerial image 12 is observed when the orientation of the person's face detected by the acquisition unit 98 is the direction facing the aerial image 12. If the face orientation is other than that, it is determined that the aerial image 12 is not observed.
And when the observation determination part 96 determines with observing the aerial image 12, it performs the process which measures the observation time.
As described above, the imaging control unit 94 performs imaging by the camera 5 in a short period regardless of whether the aerial image 12 is displayed. On the other hand, the observation determination unit 96 determines the presence or absence of observation of the aerial image 12 every time this repeated shooting is performed, and determines that there is no observation from the shooting where it is determined that there is observation and observation by a person is started. The observation period is measured by integrating the shooting period until the shooting is performed.
The imaging control unit 94, when measuring the observation time, inputs the observation time to the acquisition unit 98, and the acquisition unit 98 associates with the display image at that time as one of the state information of the person, and stores the state information storage unit. 99.

[Control of signage device by control device]
Control of the signage device 10 by the control device 9 having the above configuration will be described based on the flowchart of FIG.
In the flowchart shown in FIG. 10, only when the facial expression of the person detected from the photographed image by the acquisition unit 98 is a smile or a surprise, the display image displayed at that time is associated with which display image. A case where the facial expression is stored as the state information of the person is illustrated.

First, the imaging control unit 94 of the control device 9 puts the light emitting unit 7 into an infrared light emitting state, and starts periodic imaging with the camera 5 (step S1).
Then, the person detection unit 95 detects a person and its face reflected in the captured image. Based on this detection, it is determined whether or not there is a person in the shooting range (step S3).

  As a result, if there is no person in the shooting range (step S3: NO), the process proceeds to step S21. If it is determined that there is a person in the shooting range (step S3: YES), the first display Based on the control unit 91, the display image 11 is displayed by the image display device 1, and an aerial image 12 is also formed accordingly (step S5).

Next, the acquisition unit 98 detects various state information from the captured image. At this time, from the orientation of the face of the person detected as one of the state information based on the face of the person detected in the captured image, the observation determination unit 96 determines that the orientation of the face is the front side, that is, the aerial image. It is determined whether or not the face is directed to 12 (step S7).
As a result, when the face is directed to the aerial image 12 (step S7: YES), the time during which the face is directed to the aerial image 12 is measured (step S9). That is, the shooting period is integrated.

  On the other hand, when the face is not directed to the aerial image 12 (step S7: NO), the shooting cycle is not accumulated, and when the face is directed to the aerial image 12 until the last shooting, The integrated value of the time when the face has been directed to the aerial image 12 so far is added to one of the state information as a measurement time and stored in the state information storage unit 99. Then, the process proceeds to step S19, the display is switched to the display image of the next content based on the second display control unit 92, and the process returns to step S1.

Subsequent to step S9, the acquisition unit 98 determines whether the facial expression of the person detected as one of the state information is a smile (step S11).
If the detected facial expression of the person is a smile (step S11: YES), the detected person's face is displayed in association with which content display image is displayed. The facial expression is stored in the state information storage unit 99 as one piece of state information (step S13).
Then, the process proceeds to step S19, the display is switched to the display image of the next content based on the second display control unit 92, and the process returns to step S1.

If the detected facial expression of the person is not a smile (step S11: NO), it is determined whether the detected facial expression of the person is surprising (step S15).
If the detected facial expression of the person is surprising (step S15: YES), the content of the detected person is displayed in association with which content display image is displayed. The facial expression is stored in the state information storage unit 99 as one of the state information (step S17).
Then, the process proceeds to step S19, the display is switched to the display image of the next content based on the second display control unit 92, and the process returns to step S1.

  If the detected facial expression of the person is not surprising (step S15: NO), the process proceeds to step S19 as it is, and the display image of the next content is displayed based on the second display control unit 92. The display is switched to and the process returns to step S1.

In addition, in step S3 described above, when there is no person in the shooting range (step S3: NO), the first display control unit 91 determines in advance the elapsed time from the absence of the person in the shooting range. It is determined whether the specified time has elapsed (step S21).
If the specified time has not elapsed since the person is no longer in the shooting range (step S21: NO), the process returns to step S1, and the next shooting is performed.
In addition, when the specified time has passed since the person is no longer in the shooting range (step S21: YES), if the display image 11 is being displayed by the image display device 1, it is stopped (step S21). S23), the process is returned to step S1, and the next shooting is performed.

[Effects of each signage device configuration]
When the display image 11 is displayed on the image display device 1 based on predetermined display image data in the signage device 10 having the above configuration, the light from the image display device 1 is incident on the incident surface 22 of the optical plate 2, The light is transmitted through the beam splitter 4 from the exit surface 23 side.
As a result, the aerial image 12 is formed at a position that is plane-symmetric with respect to the optical plate 2. Similar to the display image 11, the aerial image 12 is an image based on predetermined display image data.
Along with the display of the display image 11 by the image display device 1, the LED emits light in the light emitting unit 7, and infrared light is irradiated toward the prescribed observation position K with respect to the aerial image 12.
At this time, if there is a person within the shooting range, infrared light is irradiated, and infrared scattered light irradiated to the person is generated.
Since the camera 5 is directed so that the optical axis 5a through the beam splitter 4 is perpendicular to the aerial image 12, infrared scattered light from a person is photographed from the vertical direction with respect to the aerial image 12. .
At this time, since the transflective layer 41 is formed on the beam splitter 4, most of the scattered infrared light generated in the person is reflected to the camera 5 side without passing through the beam splitter 4.
Further, most of the visible light around the person is transmitted through the semi-transmissive layer 41 of the beam splitter 4, and a part of the reflected visible light is also cut by the filter member 51 of the camera 5, and most of the visible light is not photographed.
As described above, the signage device 10 captures the infrared scattered light generated in the person from the direction perpendicular to the aerial image 12 in a state where the visible light around the person is sufficiently eliminated, and based on this, A person can be photographed satisfactorily, and the presence / absence of the person and state information of the person can be detected with high accuracy.

[Technical effects of the first embodiment]
In the signage device 10, the camera 5 captures an imaging range including an observation position farther than the aerial image 12 through the beam splitter 4 that transmits a part of the light from the display image 11 and reflects the other part. doing.
Therefore, when there is a person in the shooting range, the infrared scattered light of the person can be taken from the direction perpendicular to the aerial image 12, reducing the influence of trapezoidal distortion, etc. A good captured image for performing detailed detection can be obtained, and various state information can be detected with high accuracy.

Further, since the signage device 10 includes the light emitting unit 7 that emits infrared light with respect to the shooting range, the influence of visible light around the person is influenced based on the scattered light generated when the person exists within the shooting range. Therefore, the person can be photographed, and various state information about the person can be detected with higher accuracy.
In particular, by matching the irradiation range H of the infrared light by the light emitting unit 7 with the size of the shooting range S of the camera 5 at the specified observation position K with respect to the aerial image 12, the infrared ray is not irradiated to an unnecessary range, and the shooting is performed. It is possible to detect various state information related to persons within the range with higher accuracy.

  In addition, since the camera 5 captures images through the filter member 51 that cuts visible light, when a person is present in the capturing range, the influence of visible light around the person can be reduced to capture the person. It is possible to further improve the detection accuracy of various state information.

  In addition, since the beam splitter 4 includes the semi-transmissive layer 41 that transmits visible light on the surface and reflects infrared light that is invisible, the influence of visible light around the person can be further reduced. It becomes possible to further improve the position detection accuracy of the state information.

In addition, the signage device 10 displays a person detection unit 95 that detects a person included in the shooting range from the photographed image of the camera 5 by the control device 9, and displays the aerial image 12 when a person is detected by the person detection unit 95. And a first display control unit 91 that terminates the display of the aerial image 12 after a certain period of time has elapsed when no person is detected by the person detection unit 95.
For this reason, it is possible to suppress the display of the aerial image 12 which is unnecessary when there is no person in the surroundings, and it is possible to perform appropriate display. In addition, a power saving effect can be obtained.

Further, the control device 9 includes an observation determination unit 96 that determines whether or not the aerial image 12 is observed from the orientation of the face of the person included in the captured image of the camera 5, and the acquisition unit 98 includes the observation determination unit 96. When it is determined that the aerial image 12 is being observed, processing for measuring the observation time is performed as the state information of the person.
Therefore, by obtaining the observation time, it is possible to obtain information for determining how much the person is interested in the display image (aerial image 12) at that time, and obtain useful information for the display image. It becomes possible. In particular, when the display image is an advertisement or the like, it is possible to research the advertising effect and the degree of attention by the advertisement.

In addition, the control device 9 includes a second display control unit 92 that sequentially displays a plurality of display images on the image display device 1, and the acquisition unit 98 detects and detects a facial expression of a person included in the captured image of the camera 5. When the displayed facial expression is a prescribed facial expression, the state information storage unit 99 stores the facial expression of the person as the state information of the person by associating which one of the plurality of contents is the display image displayed at that time. Processing to memorize is performed.
For this reason, by associating the facial expression of the person with the type of content, information on how the person is interested in the display image (aerial image 12) at that time, and whether the reputation is good or bad is determined. It is possible to obtain useful information for the display image. In particular, when the display image is an advertisement or the like, it is possible to research the advertising effect, favorableness, reputation, and the like by the advertisement.

In addition, the image display apparatus 1 includes a second display control unit 92 that sequentially displays the display images related to the plurality of contents, and the acquisition unit 98 classifies the display images of the plurality of contents for each person to acquire the state information of the person. , Processing to be stored in the state information storage unit 99 is performed.
For this reason, it is possible to effectively collect information for determining the degree of interest and the degree of attention for each content.
In particular, when the display image is an advertisement or the like, it is possible to effectively research information determined by the advertisement such as an advertising effect, favorableness, reputation, and attention.

Further, in the signage device 10, as an optical element that forms an aerial image by forming light from a planar object in the air, a plurality of reflecting surfaces 212 that are orthogonal to each other when viewed from the direction perpendicular to the plate plane. , 312 is used.
Thereby, it becomes possible to form the aerial image 12 more finely.

Here, the influence on the resolution when the direction in which the camera 5 captures the image is inclined with respect to the direction normal to the aerial image 12 will be described.
FIG. 11 shows a case where the optical axis 5a of the camera 5 is oriented perpendicularly to the imaging plane (when facing) and the optical axis 5a of the camera 5 is inclined at an angle θ with respect to the normal of the imaging plane. It is explanatory drawing which shows the dimension of each part in the case of inclining (it shall be in an inclined state).

The symbols in FIG. 11 are as follows.
ω: Half angle of view of camera 5 T: Distance from camera 5 to imaging plane E Da: Width of imaging range of camera 5 in the face-to-face state along imaging plane E Db: Imaging range of camera 5 in tilted state Width a in the direction along the shooting plane E: Width in the direction along the shooting plane E from one end of the shooting range of the camera 5 in the facing state to one end of the shooting range in the inclined state b: In the facing state Width in the direction along the photographing surface E from the other end of the photographing range of the camera 5 to the other end of the photographing range in the inclined state

The following equations (1) to (4) are established for the above dimensions.
a = Ttanω−Ttan (ω−θ) = T {tanω−tan (ω−θ)} (1)
b = Ttan (ω + θ) −Ttanω = T {tan (ω + θ) −tanω} (2)
Db = Da−a + b = Da−2Ttan ω + Ttan (ω−θ) + Ttan (ω + θ) (3)
Da = 2Ttanω (4)

The reduction rate R of resolution when the camera 5 is tilted from the directly-facing state to the tilted state is obtained by the following equation (5). Note that these calculations are approximate without considering the distortion of the lens of the optical system of the camera 5.
R = Da / Db
= 2Ttanω / {Ttan (ω−θ) + Ttan (ω + θ)}
= 2tanω / {tan (ω−θ) + tan (ω + θ)} (5)

According to the above equation (5), for example, when the optical axis 5a of the camera 5 is photographed from a direction inclined by 30 ° with respect to the normal line of the aerial image 12, the resolution reduction rate R = 66.7%.
Therefore, when the image is taken from the direction in which the optical axis 5a of the camera 5 is perpendicular to the aerial image 12 via the beam splitter 4 as in the signage device 10, not only the influence of trapezoidal distortion but also the resolution due to the inclination is obtained. It is possible to effectively suppress the reduction, and more accurately determine the presence / absence of the person at the observation position and detect the person's state information.

[Addition of total reflection mirror]
FIG. 12 shows an example in which a total reflection mirror 52 is added to the signage device 10 (illustration is omitted for some configurations). In this example, since the total reflection mirror 52 is newly added and the arrangement of the camera 5 is the same as that of the signage device 10 described above, the other components are denoted by the same reference numerals and redundant description is omitted. .

In the signage device 10, it is necessary to arrange the camera 5 while avoiding the light path from the display image 11 through the optical plate 2 to the aerial image 12. It was arranged away from.
Therefore, a total reflection mirror 52 is provided in such a manner that the reflection surface faces the beam splitter 4, the optical axis 5 a of the camera 5 is reflected toward the beam splitter 4 by the total reflection mirror 52, and the light reflected on the beam splitter 4 is further reflected. The camera 5 is arranged so that the axis 5a is perpendicular to the aerial image 12.

As a result, the camera 5 can be disposed closer to the beam splitter 4 and further closer to the image display device 1 than the beam splitter 4, and the signage device 10 can be downsized.
This is also effective when the space for arranging the camera 5 cannot be secured above the beam splitter 4.

The total reflection mirror 52 may be arranged such that the reflection surface is parallel to the XY plane. However, if the optical axis 5a of the camera 5 is perpendicular to the aerial image 12, total reflection is performed. The reflecting surface of the mirror 52 may be arranged to be inclined with respect to the XY plane.

[Second Embodiment]
A signage device 10A according to a second embodiment of the present invention will be described with reference to FIGS. FIG. 13 is an explanatory diagram schematically showing the main configuration of the signage device 10A.
This signage apparatus 10A is characterized in that the retroreflector 2A is used as an optical element that forms an aerial image 12 by forming light from a planar object (display image 11) in the air.
The structure of the signage device 10A is the same as that of the signage device 10 described above except that the optical plate 2 is replaced with the retroreflector 2A. Omitted. In FIG. 13, the casing 8 and the control device 9 are not shown.

The retroreflector 2A is arranged on the display image 11 side of the beam splitter 4 with the retroreflective surface 21A facing the beam splitter 4 side.
The retroreflective surface 21A of the retroreflector 2A has retroreflectivity, and has a characteristic of emitting light incident on the retroreflective surface 21A in a direction opposite to and parallel to the incident direction by internal reflection.
On the reflection surface, glass beads and microprisms made of microspheres are spread all over, and light incident on the glass beads or microprisms can be emitted in parallel and in opposite directions by internal refraction and reflection. . Since the retroreflector has a well-known structure, detailed description thereof is omitted.

The retroreflector 2A is arranged such that light from the display image 11 is reflected by the beam splitter 4 and enters the retroreflective surface 21A.
As shown in FIG. 14, the retroreflector 2 </ b> A is configured such that any light emitted from the point d of the display image 11 toward the beam splitter 4 in each direction is retroreflected by the retroreflector 2 </ b> A. The surface 21A reflects in the opposite direction parallel to the incident direction. As a result, the light emitted from the point d of the display image 11 passes through the beam splitter 4 and forms an image at a point d ₀ that is plane-symmetric with respect to the beam splitter 4.
Since all the points d located in any of the display images 11 are similarly imaged, the display image 11 is imaged at a position that is plane-symmetric with respect to the beam splitter 4 to form an aerial image 12.

  The retroreflecting surface 21A of the retroreflector 2A is preferably inclined so as to face the beam splitter 4 and the display image 11 as shown in FIG. 13, but the beam splitter 4 is perpendicular to the display image 11. If the light reflected by the beam splitter 4 is directed to the retroreflecting surface 21A of the retroreflector 2A as a whole, the tilt angle can be changed.

Similar to the signage device 10 described above, the signage device 10A reduces the trapezoidal distortion because the camera 5 is arranged such that the optical axis 5a through the beam splitter 4 is perpendicular to the aerial image 12. In addition, it is possible to suppress a decrease in resolution during photographing. Therefore, even when the retroreflector 2A is used, as in the case of the optical plate 2, the presence / absence determination of the person at the observation position and the detection of the person state information can be performed with higher accuracy.
Furthermore, since the signage device 10A includes the retroreflector 2A as an optical element, the signage device 10A is relatively easy to manufacture as compared to the optical plate 2, and the manufacturing cost can be reduced.
Further, since the retroreflector 2A can set the inclination angle of the retroreflective surface 21A in a wider range, the degree of freedom in designing the arrangement of the internal configuration of the apparatus is high, and the apparatus can be easily downsized. Can be realized.

[Other examples of arrangement of camera and retroreflector]
FIG. 15 is an explanatory view showing another example of the arrangement of the camera 5 and the retroreflector 2A inside the apparatus.
In this example, the arrangement of the image display device 1 with respect to the beam splitter 4 is the same as the arrangement of the signage device 10A in FIG. 13, but the arrangement of the camera 5 is the surface of the beam splitter 4 with respect to the arrangement of the signage device 10A in FIG. The positions of the retroreflectors 2A are changed to positions that are symmetrical with respect to the beam splitter 4 with respect to the arrangement of the signage device 10A of FIG.
Further, in the example of FIG. 15, the semi-transmissive layer 41 is not formed on the beam splitter 4.

When the camera 5 and the retroreflector 2A are arranged as shown in FIG. 15, when the display image 11 is displayed on the image display device 1, the light from the image display device 1 is transmitted through the beam splitter 4 and retroreflected. The retroreflected surface 21 </ b> A of the body 2 </ b> A is retroreflected, and the reflected light is reflected by the beam splitter 4 to form an aerial image 12 at a position that is plane-symmetric with the display image 11 with respect to the beam splitter 4.
On the other hand, since the camera 5 is arranged so that the optical axis passes through the beam splitter 4 and is perpendicular to the aerial image 12, the imaging range and the person observing the aerial image 12 can be captured from the front. Can do.
Therefore, in the case of the arrangement shown in FIG. 15 as well as the signage device 10A in FIG. 13, it is possible to reduce the trapezoidal distortion and to suppress the reduction in resolution at the time of shooting, and to determine the presence or absence of a person within the shooting range. It is possible to detect the state information of the person and the person more accurately.

[Examples of using stereo cameras]
In addition, the signage devices shown in FIGS. 1, 12, 13, and 15 described above have exemplified the configuration in which shooting is performed by a single camera 5. However, as shown in FIG. A stereo camera 5B may be used.
FIG. 16 illustrates a case where the camera 5 of the signage device 10 of FIG. 1 is changed to a stereo camera 5B.

  The stereo camera 5B is composed of two cameras 55B and 56B, both of which are arranged such that the optical axis through the beam splitter 4 is perpendicular to the aerial image 12, and at least each of the front side of the beam splitter 4 They are spaced apart in any direction perpendicular to the optical axis. Here, the case where the cameras 55B and 56B are spaced apart from each other in the X direction is illustrated.

In the case of the configuration in which the stereo camera 5B is provided, the control device of the signage device uses the parallax between the two cameras 55B and 56B so that each pixel of the captured images of the camera 55B and the camera 56B is on the optical axis. Can be specified.
Therefore, by setting a general distance for observing the aerial image 12 and determining a moving object photographed within the distance range as a person, it is possible to effectively and accurately detect a person observing the aerial image 12. It becomes possible.

Here, the case where the stereo camera 5B is applied to the signage device 10 of FIG. 1 has been described as an example. However, when the stereo camera 5B is applied to the signage device shown in FIGS. The person to be observed can be effectively and accurately detected.
Furthermore, even when the stereo camera 5B is applied to any of the signage devices shown in FIGS. 1, 12, 13, and 15, each camera 55B and 56B has an optical axis in a direction perpendicular to the aerial image 12. Therefore, it becomes possible to reduce trapezoidal distortion and to accurately detect the state information of a person.

[Others]
In addition, the details shown in the various embodiments described above can be changed as appropriate without departing from the spirit of the invention.
For example, in any signage device, the filter member 51 is provided in the camera 5 and the semi-transmissive layer 41 is formed in the beam splitter 4, but only one of these may be provided.

  In addition, as the status information of the person, the pupil part of the eyeball is further detected from the detected face of the person, thereby identifying the direction that the person is observing and the display content ahead of the line of sight, Any of these may be stored as state information.

DESCRIPTION OF SYMBOLS 1 Image display apparatus 11 Display image 12 Aerial image 2 Optical plate (optical element)
2A retroreflector (optical element)
20, 30 Optical panel 21, 31 Plate member 21A Retroreflective surface 212, 312 Reflective surface 4 Beam splitter 41 Semi-transparent layer 5 Camera 5B Stereo camera 5a Optical axis 51 Filter member 52 Total reflection mirror 55B, 56B Camera 7 Light emitting part 9 Control device 91 First display control unit 92 Second display control unit 93 Image storage unit 94 Shooting control unit 95 Person detection unit 96 Observation determination unit 98 Acquisition unit 99 State information storage unit 10, 10A Signage device

Claims (12)

An optical element that forms an aerial image by focusing light from an image display device that displays a display image in the air; and
A beam splitter that transmits part of the light from the image display device and reflects the other part;
A camera arranged such that an optical axis passing through the beam splitter is perpendicular to the planar aerial image;
An acquisition unit that acquires state information of a person included in an image captured by the camera;
A signage device comprising: a state information storage unit that stores state information of the person acquired by the acquisition unit.   The signage device according to claim 1, further comprising: a light emitting unit that emits invisible light toward a predetermined observation position with respect to the aerial image.   The signage device according to claim 2, wherein an irradiation range of the invisible light by the light emitting unit is adjusted to a size of a shooting range of the camera at a predetermined observation position with respect to the aerial image.   The signage device according to claim 2 or 3, wherein the camera performs photographing through a filter member that cuts visible light.   The signage device according to any one of claims 2 to 4, wherein the beam splitter includes a semi-transmissive layer that transmits visible light and reflects invisible light on a surface thereof.   The signage device according to any one of claims 1 to 5, wherein the camera is a stereo camera. A person detection unit for detecting a person included in the shooting range of the camera from the shot image of the camera;
A first display control unit that displays the aerial image when the person is detected by the person detection unit, and terminates the display of the aerial image after a certain period of time when the person is no longer detected by the person detection unit; The signage device according to any one of claims 1 to 6, further comprising: An observation determination unit for determining whether or not the aerial image is observed from the orientation of a person's face included in the photographed image of the camera;
The said acquisition part measures the observation time as said person's state information, when it determines with the said aerial image being observed by the said observation determination part, The any one of Claim 1 to 7 characterized by the above-mentioned. A signage device according to claim 1. A second display control unit that sequentially displays a plurality of display images on the image display device;
The acquisition unit detects a facial expression of a person included in an image captured by the camera, and when the detected facial expression is a specified facial expression, the display image displayed at that time is one of the plurality of display images. The signage device according to any one of claims 1 to 8, wherein the state information storage unit stores the facial expression of the person as the state information of the person in association with each other. A second display control unit that sequentially displays a plurality of display images on the image display device;
The signage device according to any one of claims 1 to 9, wherein the acquisition unit classifies each of the plurality of display images to acquire state information of the person.   The signage device according to any one of claims 1 to 10, wherein the optical element is an optical plate having a plurality of reflecting surfaces orthogonal to each other when viewed from a direction perpendicular to the plate plane.   The signage device according to any one of claims 1 to 10, wherein the optical element is a retroreflector having a retroreflective surface that reflects along an incident direction of incident light.

Images