KR102269088B1 - Apparatus and method for tracking pupil - Google Patents

Abstract

A pupil tracking device and method are disclosed. The pupil tracking device includes an image acquisition unit that acquires an image of the object by photographing an object, a spatial position detection unit that detects a three-dimensional position of a specific part within the object from the image, and a hologram of the three-dimensional and a display unit that outputs to a space corresponding to the location.

Description

Pupil tracking device and method {APPARATUS AND METHOD FOR TRACKING PUPIL}

An embodiment of the present invention relates to a technique for solving the limited viewing area of a tabletop type digital hologram display by tracing the position of a pupil corresponding to a space to output a hologram using an image of an object obtained from a different angle.

Digital holography display technology is a technology that outputs a stereoscopic image in a three-dimensional space using a light diffraction phenomenon such as a laser based on a light modulator (SLM). The tabletop type holographic display refers to a display that outputs an image on a space on a flat table using holographic display technology so that a stereoscopic image can be viewed from any angle of 360 degrees.

1 is a diagram illustrating the principle that an image appears to float in space by reflecting a laser on a spherical mirror, and FIG. 2 is a diagram illustrating the principle that an image appears to float in space by reflecting light from the human eye.

Referring to FIGS. 1 and 2 , the holographic display adjusts the direction of the laser light diffraction angle based on the optical effect using a spherical mirror, thereby giving an effect like an image floating in space. That is, the holographic display projects the image diffracted in the direction the human eye sees in various directions in the air to give the effect as if floating in the space.

As the tabletop holographic display is based on an optical modulator and laser like the holographic display, the hologram observation range is limited due to the pixel size of the optical modulator.

In order to compensate for this limitation, a method of controlling the direction of light output through a laser or a light modulator according to the position of a person's pupil is required, and for this, a technology for accurately tracking a person's pupil in a three-dimensional space is required.

An embodiment of the present invention aims to solve the limited viewing area of a tabletop type digital hologram display by accurately tracing the position of the pupil corresponding to the space to output the hologram using images of objects obtained from different angles. .

A pupil tracking apparatus according to an embodiment of the present invention includes an image acquisition unit that captures an object to acquire an image of the object, and a spatial position detection unit that detects a three-dimensional position of a specific part within the object from the image; , a display unit for outputting a hologram in a space corresponding to the three-dimensional position.

The image acquisition unit may include n (n is a natural number) cameras positioned in different orientations.

The camera may be any one of a stereo camera, a color camera, or a depth camera.

The spatial position detection unit may include a pupil tracking unit that determines whether a pupil is included as the specific part in the image, and tracks a two-dimensional position with respect to the pupil when it is determined that the pupil is included. .

The pupil tracking unit may divide the image into a plurality of set regions, track a region having the largest average luminance value as an eye region in the face, and track the pupil in the eye region.

The spatial position detector may further include a pupil position calculator configured to calculate a three-dimensional position with respect to the pupil, based on a two-dimensional position with respect to the pupil and state information about a camera in the image acquisition unit.

The image acquisition unit may include an omnidirectional camera for photographing the object in all directions, and a plurality of panoramic cameras for photographing the object in different directions so as to acquire a panoramic image for all directions.

The pupil tracking device selects at least one panoramic camera from among the plurality of panoramic cameras based on the identification information about the camera received from the omnidirectional camera, receives an image from the selected panoramic camera, and transmits it to the spatial position detection unit It may further include an image selection unit.

When the object is extracted from the image, the omnidirectional camera may provide identification information about the camera corresponding to the direction in which the object is located to the image selection unit.

The pupil tracking method according to an embodiment of the present invention includes the steps of acquiring an image of the object by photographing an object, detecting a three-dimensional position of a specific part within the object from the image, and generating a hologram. and outputting it to a space corresponding to a three-dimensional position.

According to an embodiment of the present invention, the limited viewing area of the tabletop digital hologram display can be solved by accurately tracking the position of the pupil corresponding to the space where the hologram is to be output using the image of the object obtained from different angles.

1 is a view showing the principle that an image appears to float in space by reflecting a laser on a spherical mirror, and FIG. 2 is a view showing the principle that an image appears to float in space by reflecting light from the human eye.
Figure 3 is a diagram showing an example of the configuration of the pupil tracking device according to an embodiment of the present invention.
Figure 4 is a view showing an example of the arrangement of the camera in the pupil tracking device according to an embodiment of the present invention.
5 is a view for explaining an example of tracking the position of the pupil in the pupil tracking device according to an embodiment of the present invention.
6 is a diagram illustrating an example of the configuration of a pupil tracking device according to another embodiment of the present invention.
7 and 8 are diagrams showing an example of the arrangement of the camera in the pupil tracking device according to another embodiment of the present invention.
9 is a flowchart illustrating a pupil tracking method according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings and the contents described in the accompanying drawings, but the present invention is not limited or limited by the embodiments.

Figure 3 is a diagram showing an example of the configuration of the pupil tracking device according to an embodiment of the present invention.

Referring to FIG. 3 , the pupil tracking apparatus 300 according to an embodiment of the present invention may include an image acquisition unit 301 , a spatial position detection unit 303 , and a display unit 309 .

The image acquisition unit 301 may include n cameras (n is a natural number) positioned in different orientations, and acquire an image of the object by photographing an object (eg, a person) through the n cameras. can do. Here, the image acquisition unit 301 may include two stereo cameras positioned in each direction or a camera (eg, a depth camera) that recognizes an object as 3D information (eg, depth information and color information). .

The n cameras in the image acquisition unit 301 may be connected to n pupil tracking modules in the pupil tracking unit 305 , respectively, to transmit each image of the object to the n pupil tracking modules.

The spatial position detection unit 303 may detect a three-dimensional position of a specific part (eg, a pupil) within the object from the image received from the image acquisition unit 301 . Here, the spatial position detecting unit 303 may include a pupil tracking unit 305 and a pupil position calculating unit 307 .

The pupil tracking unit 305 may include n pupil tracking modules (eg, first to nth pupil tracking modules). Here, the pupil tracking module may receive an image from the camera and determine whether a pupil is included in the received image. Here, the pupil tracking module may divide the image into a plurality of set regions, track an eye region within the face in an region where a human face is determined to exist, and track the pupil in the eye region. In this case, the pupil tracking module may track a region having the largest output value passing through the pupil classifier among the plurality of divided regions as an eye region in the face.

That is, when it is determined that the pupil is included in the image, the pupil tracking module first finds the face in the image, finds the position of the eye within the extracted face, and finds the pupil (eg, left, right) within the position of the corresponding eye. Pupil) can be tracked in two dimensions (ie, in two dimensions).

The pupil tracking module may transmit the two-dimensional position of the tracked pupil to the pupil position calculator 307 .

The pupil position calculation unit 307 receives the two-dimensional positions of the pupils from the n pupil tracking modules in the pupil tracking unit 305, respectively, and state information about the received two-dimensional positions and n cameras (for example, , information about the position, angle, and direction of the camera, and the resolution of the camera), a three-dimensional position with respect to the pupil (ie, a position on a three-dimensional space) may be calculated.

At this time, the pupil position calculation unit 307 receives the respective two-dimensional positions for the left and right pupils from the n pupil tracking modules in the pupil tracking unit 305, and the received two-dimensional for the n left and right pupils. 3D positions for the left and right pupils can be calculated using the positions of . Here, when the image is acquired by two stereo cameras, the pupil position calculator 307 uses a disparity between the two stereo cameras positioned in succession to determine the distance between the left and right pupils and the three-dimensional space. position can be calculated.

The display unit 309 may output the generated hologram in a space corresponding to a three-dimensional position (eg, a three-dimensional position with respect to the pupil) of the specific portion detected by the spatial position detection unit 303 .

Figure 4 is a view showing an example of the arrangement of the camera in the pupil tracking device according to an embodiment of the present invention.

Referring to FIG. 4 , the pupil tracking apparatus may acquire a plurality of images by photographing an object (eg, a person) through a plurality of cameras (camera arrays) respectively located in different orientations. Here, the plurality of cameras may be located spaced apart along a virtual circle, for example.

5 is a view for explaining an example of tracking the position of the pupil in the pupil tracking device according to an embodiment of the present invention.

Referring to FIG. 5 , the pupil tracking device may track the pupil by, for example, calculating the position of the pupil using a Haar feature-based approach.

Here, in the Haar feature-based approach, the Haar feature is configured as one filter set, the response to each filter is configured as a classifier using a face database, and the input image is It is an approach that compares the output value passed through with a threshold, and determines whether or not a face is present based on the comparison result.

When an image is input, the pupil tracking apparatus may detect candidate regions of faces or eyes in units of various sizes. In this case, the pupil tracking device can detect the size unit having the largest output value (eg, the largest output value of the Har classifier) output through the pupil classifier among several size units from the image as the face or eye region. have. The pupil classifier may be a means for numerically or probabilistically evaluating a region estimated to be a pupil in an input image, and is applied to the pupil tracking apparatus of the present invention, and an output value for each region divided in association with the image can be evaluated. The pupil tracking apparatus may compare the output values evaluated by the pupil classifier in each of a plurality of regions, and may track the region evaluated as the largest output value among the plurality of regions as the region of the eye in the face.

When the position of the eye is detected, the pupil tracking device may detect the position of the pupil based on the center of the eye. At this time, when detecting the position of the pupil based on the position of the eye, the pupil tracking device detects the position of the pupil based on the fact that the pupil comes out darker in the image acquired by the camera and the shape of the pupil is close to a circle. can

In this case, the pupil tracking apparatus may detect the position of the pupil by using a circular detection algorithm based on Equation (1).

Here, I(x, y) denotes the pixel value at the (x, y) position, (x0, y0) denotes the center of the circle, and r denotes the radius.

Specifically, the pupil tracking device adds all the pixel values around the circle normalized to 2πr by the radius (r) from the center (x0, y0) of the circle through [Equation 1], and The circumference of the circle when the difference between the respective pixel values around the circle is the largest may be determined as the pupil region. In this case, the pupil tracking apparatus may increase the accuracy of pupil detection by performing a Gaussian function (G(r)) in the direction of the radius (r) when detecting the circumference of a circle in order to remove noise.

6 is a diagram illustrating an example of the configuration of a pupil tracking device according to another embodiment of the present invention.

Referring to FIG. 6 , the pupil tracking apparatus 600 according to another embodiment of the present invention may include an image acquisition unit 601 , an image selection unit 603 , a spatial position detection unit 605 , and a display unit 611 . can

The image acquisition unit 601 may include a plurality of cameras. Here, the plurality of cameras take one omni-directional camera capable of photographing omnidirectional (that is, 360 degrees) at once, and objects in different directions to obtain panoramic images for all directions. Thus, it may be composed of a plurality of panoramic cameras each acquiring an image.

At this time, the omnidirectional camera extracts an object (eg, a person) from the captured image 613 , and when the object is extracted, the identification information (or location information of the camera) related to the object is provided to the image selection unit 603 . can be forwarded to Here, the camera associated with the object may be a camera corresponding to the orientation in which the object is located.

The image selection unit 603 may receive some images among the plurality of images acquired by the image acquisition unit 601 . In this case, the image selection unit 603 may include, for example, a camera switch, and selects at least one panoramic camera from among a plurality of panoramic cameras based on the identification information about the camera received from the omnidirectional camera, By switching on the selected panoramic camera, an image may be received from the panoramic camera.

Here, the image selection unit 603 has been described as receiving the identification information about the camera from the omnidirectional camera, but is not limited thereto, and receives the image from the omnidirectional camera, and obtains the identification information about the camera from the received image can do.

The spatial position detector 605 may detect a three-dimensional position of a specific part (eg, a pupil) within the object from at least one image received from the image selection unit 603 . Here, the spatial position detecting unit 605 may include a pupil tracking unit 607 and a pupil position calculating unit 609 .

The pupil tracking unit 607 may include a plurality of pupil tracking modules. The pupil tracking unit 607 may receive at least one image from the image selection unit 603 and determine whether a pupil is included in the received image. Here, the pupil tracking module divides the image into a plurality of set regions, and selects a region having the largest output value of a pupil classifier (eg, a Haar feature-based classifier) among the divided regions, for example, an eye region within a face. It is possible to track the pupil in the region of the eye.

When it is determined that the pupil is included in the image, the pupil tracking module may track the two-dimensional position of the pupil (eg, left and right pupil) and transmit it to the pupil position calculator 609 .

The pupil position calculation unit 609 receives the two-dimensional position of the pupil from the plurality of pupil tracking modules in the pupil tracking unit 607, respectively, and state information (eg, the received two-dimensional position and the plurality of cameras) Based on information about the position, angle, and direction of the camera), a three-dimensional position with respect to the pupil may be calculated.

The display unit 611 may output the generated hologram in a space corresponding to a three-dimensional position with respect to the pupil.

The pupil tracking device 600 according to another embodiment of the present invention detects a pupil using an image obtained from an effective camera (eg, some selected panoramic camera) selected by an omnidirectional camera, thereby reducing the amount of calculation for pupil tracking , it is possible to efficiently track the position relative to the pupil.

7 and 8 are diagrams showing an example of the arrangement of the camera in the pupil tracking device according to another embodiment of the present invention. 7 is a camera layout when viewed from the front, and FIG. 8 is a camera layout when viewed from above.

Referring to FIGS. 7 and 8 , the pupil tracking device includes one omni-directional camera capable of photographing 360 degrees at once, and a plurality of positions each positioned to obtain a panoramic image for 360 degrees. A plurality of images may be acquired by photographing an object (eg, a person) through a panoramic camera (camera arrays). Here, when viewed from the front, the plurality of panoramic cameras, for example, may be located lower than the omnidirectional camera, and when viewed from above, the plurality of panoramic cameras are spaced apart from the omnidirectional camera, for example, along a virtual circle. can do.

9 is a flowchart illustrating a pupil tracking method according to an embodiment of the present invention.

Referring to FIG. 9 , in step 901 , the pupil tracking apparatus may photograph an object to acquire an image of the object.

In this case, the pupil tracking apparatus may acquire an image of the object by using n (n is a natural number) cameras located in different orientations. Here, the camera may be any one of a stereo camera, a color camera, or a depth camera.

In operation 903, the pupil tracking apparatus may detect a three-dimensional position of a specific part in the object from the image.

The pupil tracking apparatus may determine whether a pupil is included as the specific part in the image, and, if it is determined that the pupil is included, may track a two-dimensional position of the pupil. At this time, the pupil tracking device divides the image into a plurality of set regions, and among the divided regions, the region having the largest output value passing through the pupil classifier (eg, Haar feature-based classifier) is defined as the region of the eye in the face. By tracking and tracking the pupil in the region of the eye, it is possible to determine whether or not the pupil is included in the image.

Then, the pupil tracking device is based on the two-dimensional position with respect to the pupil and the state information about the camera that obtained the image (eg, information about the position, angle, and direction of the camera), the three-dimensional for the pupil location can be calculated.

In step 905, the pupil tracking device may output a hologram to a space corresponding to the three-dimensional position.

As another example, the pupil tracking device uses an omnidirectional camera for photographing the object in all directions, and a plurality of panoramic cameras for photographing the object in different directions so as to obtain a panoramic image for all directions, You can get an image for

In this case, the pupil tracking device may select at least one panoramic camera from among the plurality of panoramic cameras based on the identification information about the camera received from the omnidirectional camera. Here, when the object is extracted from the image, the pupil tracking device may receive identification information about the camera corresponding to the direction in which the object is located from the omnidirectional camera.

Thereafter, the pupil tracking apparatus may detect a three-dimensional position of a specific part within the object from the image acquired by the selected panoramic camera, and may output a hologram in a space corresponding to the three-dimensional position.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

300: pupil tracking device 301: image acquisition unit
303: spatial position detection unit 309: display unit

Claims (18)

an image acquisition unit for photographing an object to acquire an image of the object;
a spatial position detector for detecting a three-dimensional position of a specific part within the object from the image; and
A display unit for outputting a hologram to a space corresponding to the three-dimensional position,
The image acquisition unit,
Comprising an omnidirectional camera for photographing the object in all directions, and a plurality of panoramic cameras for photographing the object in different directions to obtain a panoramic image for all directions,
Obtaining a 360-degree image of the object by photographing the object in all directions through the omnidirectional camera and the plurality of panoramic cameras,
The spatial position detection unit for estimating the position of the pupil in the 360-degree image,
pupil tracking device. According to claim 1,
The image acquisition unit,
n cameras (n is a natural number) located in different orientations
A pupil tracking device comprising a. 3. The method of claim 2,
The camera is
Either a stereo camera, a color camera, or a depth camera
pupil tracking device. According to claim 1,
The spatial position detection unit,
A pupil tracking unit that determines whether a pupil is included as the specific part in the image, and tracks a two-dimensional position of the pupil when it is determined that the pupil is included.
A pupil tracking device comprising a. 5. The method of claim 4,
The pupil tracking unit,
The image is divided into a plurality of set regions, and a region having the largest output value passing through the pupil classifier among the divided regions is tracked as an eye region in the face, and the pupil is tracked in the eye region.
pupil tracking device. 5. The method of claim 4,
The spatial position detection unit,
A pupil position calculator for calculating a three-dimensional position with respect to the pupil based on the two-dimensional position with respect to the pupil and state information about the camera in the image acquisition unit
A pupil tracking device further comprising a. delete According to claim 1,
On the basis of the identification information about the camera received from the omnidirectional camera, selects at least one panoramic camera from the plurality of panoramic cameras, receives an image from the selected panoramic camera, and an image selection unit for transmitting the image to the spatial position detection unit
A pupil tracking device further comprising a. 9. The method of claim 8,
The omnidirectional camera,
When the object is extracted from the image, providing identification information about the camera corresponding to the direction in which the object is located is provided to the image selection unit
pupil tracking device. acquiring an image of the object by photographing the object;
detecting a three-dimensional position of a specific part within the object from the image; and
Comprising the step of outputting a hologram in a space corresponding to the three-dimensional position,
The acquiring of the image includes acquiring a 360-degree image of the object by photographing the object in all directions through an omnidirectional camera and a plurality of panoramic cameras,
The detecting includes estimating the position of the pupil in the 360-degree image,
The omnidirectional camera takes the object in all directions,
The plurality of panoramic cameras acquire a panoramic image for all directions by photographing the object from different orientations,
Pupil tracking method. 11. The method of claim 10,
The step of obtaining an image of the object includes:
Acquiring an image of the object by using n cameras (n is a natural number) located in different orientations
A pupil tracking method comprising a. 12. The method of claim 11,
The camera is
Either a stereo camera, a color camera, or a depth camera
Pupil tracking method. 11. The method of claim 10,
The step of detecting a three-dimensional position of a specific part in the object comprises:
determining whether a pupil is included as the specific part in the image; and
When it is determined that the pupil is included, tracking a two-dimensional position with respect to the pupil
A pupil tracking method comprising a. 14. The method of claim 13,
The step of determining whether the pupil is included,
classifying the image into a plurality of set regions, tracing a region having the largest output value passing through a pupil classifier among the plurality of divided regions as an eye region in the face, and tracing the pupil in the eye region
A pupil tracking method comprising a. 14. The method of claim 13,
The step of detecting a three-dimensional position of a specific part in the object comprises:
Calculating a three-dimensional position with respect to the pupil based on the two-dimensional position with respect to the pupil and state information about the camera that acquired the image
A pupil tracking method further comprising a. delete 11. The method of claim 10,
The pupil tracking method,
Selecting at least one panoramic camera from among the plurality of panoramic cameras based on the identification information about the camera received from the omnidirectional camera
further comprising,
The step of detecting a three-dimensional position of a specific part in the object comprises:
detecting a three-dimensional position of a specific part within the object from the image acquired by the selected panoramic camera;
A pupil tracking method comprising a. 18. The method of claim 17,
The pupil tracking method,
When the object is extracted from the image, receiving identification information about the camera corresponding to the direction in which the object is located from the omnidirectional camera
A pupil tracking method further comprising a.

Images