CN112351358B - 360-degree free three-dimensional type three-dimensional display sound box based on face detection - Google Patents
360-degree free three-dimensional type three-dimensional display sound box based on face detection Download PDFInfo
- Publication number
- CN112351358B CN112351358B CN202011212021.1A CN202011212021A CN112351358B CN 112351358 B CN112351358 B CN 112351358B CN 202011212021 A CN202011212021 A CN 202011212021A CN 112351358 B CN112351358 B CN 112351358B
- Authority
- CN
- China
- Prior art keywords
- cylindrical
- spherical lens
- module
- dimensional display
- degree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 46
- 230000003287 optical effect Effects 0.000 claims abstract description 53
- 238000003384 imaging method Methods 0.000 claims abstract description 42
- 230000005540 biological transmission Effects 0.000 claims abstract description 30
- 230000010287 polarization Effects 0.000 claims description 9
- 238000004519 manufacturing process Methods 0.000 claims description 2
- 238000003062 neural network model Methods 0.000 claims 1
- 230000002093 peripheral effect Effects 0.000 claims 1
- 239000011521 glass Substances 0.000 abstract description 4
- 230000000007 visual effect Effects 0.000 description 19
- 238000000034 method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003993 interaction Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 208000029152 Small face Diseases 0.000 description 2
- 230000004888 barrier function Effects 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000009365 direct transmission Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/02—Casings; Cabinets ; Supports therefor; Mountings therein
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B21/00—Projectors or projection-type viewers; Accessories therefor
- G03B21/14—Details
- G03B21/28—Reflectors in projection beam
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B31/00—Associated working of cameras or projectors with sound-recording or sound-reproducing means
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B35/00—Stereoscopic photography
- G03B35/18—Stereoscopic photography by simultaneous viewing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/166—Detection; Localisation; Normalisation using acquisition arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/02—Casings; Cabinets ; Supports therefor; Mountings therein
- H04R1/028—Casings; Cabinets ; Supports therefor; Mountings therein associated with devices performing functions other than acoustics, e.g. electric candles
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses a 360-degree free three-dimensional display sound box based on face detection, which comprises a 360-degree face detection module, a miniature projector module, a transmission type projection imaging optical system or a reflection type projection imaging optical system and a cylindrical holographic directional scattering screen; the 360-degree face detection module is arranged around the outer side of the three-dimensional display sound box, the cylindrical holographic directional scattering screen is arranged in the center of the three-dimensional display sound box, and the transmission type projection imaging optical system and the miniature projector module are sequentially arranged below the cylindrical holographic directional scattering screen. The 360-degree free three-dimensional display sound box based on the face detection provided by the invention has the advantages that a user can watch a 3D scene in any horizontal direction and a certain vertical direction at a proper distance without wearing special glasses, and the portability is realized.
Description
Technical Field
The invention relates to the field of three-dimensional display, in particular to a 360-degree free three-dimensional display sound box based on face detection.
Background
By the end of the 15 th century, da vinci found that people could perceive perspective and depth when viewing parallax images. The three-dimensional display modes which are popular at present are holographic three-dimensional display, parallax three-dimensional display, integrated imaging three-dimensional display, light field three-dimensional display and the like.
Holographic display is an ideal 3D display mode, but the performance of the existing equipment cannot bear the excessive information amount of holographic display, and a high-resolution and dynamic holographic image is difficult to realize.
Parallax three-dimensional display enables the left eye and the right eye of a person to see two different two-dimensional images, and parallax fusion is carried out through the brain to generate stereoscopic impression. There are two main types of parallax-type three-dimensional displays: one is to obtain stereoscopic perception by means of glasses, add extra burden to users, easily generate convergence conflict, and the display picture does not change along with the movement of observers; the other is an auto-stereoscopic three-dimensional display which can be directly observed in a naked eye state. The autostereoscopic three-dimensional display technology can provide omnidirectional parallax, and the traditional autostereoscopic display can be mainly classified into two categories: the free stereo display technology based on the cylindrical mirror array or the parallax barrier and the multi-view three-dimensional display based on the multi-projector. The display mode based on the cylindrical lens array is limited by the size of the barriers and the lens array, the system resolution is low, and the provided visual angle is small. Although the number of viewpoints is increased, the three-dimensional display technology based on multiple projectors still cannot achieve 360-degree viewing, and the system is complex, harsh in construction conditions and difficult to commercialize.
The traditional integrated imaging display system is composed of a micro lens array and a plurality of projectors, can provide full parallax display, but has high requirement on the precision of the micro lens array, small depth area of a reconstructed image and low image resolution.
The light field three-dimensional display can record and reconstruct the light rays emitted by each point element on the three-dimensional object to all directions. The light field three-dimensional display system built by people such as the Zhongqing of Zhejiang university comprises a high-speed projector and a directional scattering screen, and can provide horizontal 360-degree viewing. However, the system is high in cost, large in size and difficult to be used for civil use.
Most of sound boxes on the market only have the function of playing audio, and visual interaction experience cannot be provided. In order to design an interactive speaker with a stereoscopic display function, it is impractical to adopt the above methods in terms of economy and practicality. The HoloTube proposed by Che-Hao Hsu et al adopts a method of detecting the orientation of a person by infrared light to realize 360-degree free three-dimensional display, and simultaneously adopts a conical reflector to regulate and control a light path, light does not pass through an optical axis, and light on the same side of the reflector is imaged on the same side of a cylindrical screen. Compared with a method for directly detecting human eyes, the infrared detection precision is low, only a horizontal position can be detected, only a horizontal 360-degree visual angle can be realized, a vertical visual angle cannot be provided, the resolution ratio of the system is low, only 292 pixels exist in the vertical direction, and the display effect is not ideal.
Disclosure of Invention
The invention provides a 360-degree free three-dimensional type three-dimensional display sound box based on face detection, aiming at the problem that the existing sound box in the market can not provide three-dimensional display and interaction; the user can watch the 3D scene in any horizontal direction and a certain vertical direction at a proper distance without wearing special glasses, and the device has the advantages of portability and the like.
The invention provides the following technical scheme:
a three-dimensional display sound box of 360 degrees free stereo type based on human face detection, the three-dimensional display sound box includes 360 degrees human face detection module, miniature projector module, transmission type projection imaging optical system or reflection type projection imaging optical system and cylindrical holographic directional scattering screen; the 360-degree face detection module is arranged around the outer side of the three-dimensional display sound box, the cylindrical holographic directional scattering screen is arranged in the center of the three-dimensional display sound box, and the transmission type projection imaging optical system and the miniature projector module are sequentially arranged below the cylindrical holographic directional scattering screen.
Wherein, 360 degrees face detection modules: for detecting the face (eye) position in real time; miniature projector module: projecting an image corresponding to the 3D scene; transmissive projection optical system or reflective projection imaging optical system: projecting light rays emitted by the micro projector onto the cylindrical directional scattering screen; cylindrical directional diffusion screen: the cylindrical screen is a transmission type deflection holographic structure scattering screen, the emergent direction of incident light is modulated, the light passing through the scattering screen can be deflected in an angle, the light horizontally incident to human eyes is not scattered, the cylindrical screen is in a transparent state, and large-angle scattering is carried out in other directions, so that a user can see a complete picture at different positions.
The transmission type projection imaging optical system sequentially comprises a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens and an aspheric lens; the transmission type projection imaging optical system comprises a first cylindrical polaroid and a second cylindrical polaroid; the first cylindrical polaroid surrounds the aspheric lens, the second cylindrical polaroid surrounds the cylindrical holographic directional scattering screen, and the polarization directions of the first cylindrical polaroid and the second cylindrical polaroid are orthogonal. Namely, the polarization direction of the first cylindrical polarizer is vertical to eliminate the influence of unscattered straight light.
The third spherical lens and the fourth spherical lens form a double cemented lens, and four surfaces of the aspheric lens are aspheric surfaces.
In the transmission type projection imaging optical system, light rays sequentially pass through a first spherical mirror, a second spherical mirror, a third spherical mirror, a fourth spherical mirror and a fifth spherical mirror, are transmitted on the aspherical mirror firstly, then are internally reflected for 2 times and finally are transmitted to a first cylindrical polaroid.
The reflective projection imaging optical system sequentially comprises a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens, an aspheric lens, a sixth spherical lens, a seventh spherical lens and an aspheric reflector; the second spherical lens and the third spherical lens form a double cemented lens. The last surface of the reflecting system is an aspheric reflecting mirror, so that the system does not need to consider the influence of straight light transmission and does not need to add two polarizing plates.
The aspheric mirror in the reflective projection imaging optical system is a large-caliber (about 100mm in diameter) aspheric mirror.
In the present invention, for the transmissive structure, the final combination aspherical lens is composed of four aspherical surfaces. The light rays respectively pass through the transmission, the reflection and the transmission of the aspheric surface and finally reach the image surface, and the light rays reflected twice can pass through the optical axis and reach the image surface. For the reflective structure, the upper light emitted by the micro projector reaches the lower side of the aspheric surface reflector through the lens group, and passes through the optical axis to reach the image surface after being reflected by the aspheric surface reflector.
In the invention, for a transmission type projection imaging optical system or a reflection type projection imaging optical system, light rays in the two structures are finally transmitted or reflected through an aspherical mirror, pass through an optical axis and are imaged to the other side.
360 degree face detection module include the camera and be used for detecting the model Retinaface of people's face, the human face image input that the camera was gathered model Retinaface, output face position.
Preferably, the 360-degree face detection module comprises 4 cameras which are arranged around the circumference of the three-dimensional display sound box, wherein each camera is spaced by 90 degrees, and the vertical visual angle is 60 degrees. The method is used for detecting the position of the human face (human eyes) in real time.
Preferably, the selected camera is a distortion-free camera module, so that the detected pixel coordinate position of the human eye accurately corresponds to the actual azimuth angle of the human eye, time consumed for correcting distortion is saved, and the influence of errors caused by distortion is avoided.
Preferably, the backbone network of the model Retinaface for detecting the human face is MobilenetV2, the MobilenetV2 network is divided into stages 1, 2 and 3, the feature maps of the stages are sent to the FPN network structure to extract the shallow, middle and deep information, and the human eye position is output.
The model can improve the detection performance: the deep layer has a large receptive field, can detect a large face, the shallow layer positioning information is accurate, and a small face can be detected. The model has good robustness, and the face can be accurately identified by the model even if the two adjacent pictures are slightly misaligned at the splicing position. The face detection algorithm adopts a Retinaface model specially used for detecting small faces, the model is improved in detection accuracy compared with the traditional algorithm, and because the network is single-stage, the detection speed is very high, and the face detection method is very accurate for incomplete and unaligned faces.
The three-dimensional display sound box comprises a stepping motor and a loudspeaker module which drive the miniature projector module.
The speaker module may play music and may also provide a user experience of interacting with the 3D model.
The invention also provides a 360-degree free three-dimensional display method based on face detection, which comprises the following steps:
1) a camera in the 360-degree face detection module collects face images, the face images are spliced into a panoramic face detection image and input into a model Retinaface, and the position of a human eye is output;
2) recording the pixel value of the human eye position output in each frame in the step (1), thereby obtaining pixel intervals corresponding to the human eyes at all angles, and further calculating the pitch angle and the yaw angle of the human eyes relative to the position of the camera;
3) generating a picture of a visual angle corresponding to the human eye according to the pitch angle and the yaw angle of the human eye relative to the position of the camera in the step (2);
4) converting the picture corresponding to the visual angle in the step 3) from a Cartesian coordinate system into a polar coordinate system and carrying out pre-distortion treatment;
5) inputting the picture processed in the step 4) into a micro projector module, wherein light rays emitted by the micro projector module reach a cylindrical holographic directional scattering screen through a transmission type projection imaging optical system or a reflection type projection imaging optical system, and a 3D scene view is displayed.
In the step 1), the 360-degree face detection module comprises 4 cameras which are arranged around the circumference of the three-dimensional display sound box, wherein the interval between every two cameras is 90 degrees, and the vertical visual angle is 60 degrees; taking the smaller 480 × 360 from the resolution of each camera, and transversely splicing into a 1920 × 360 panoramic face detection image; wherein, 360 horizontal visual angles and 60 vertical visual angles are set, 5-6 pixels are allocated to each horizontal visual angle, and 6 pixels are allocated to each vertical visual angle.
In the step 2), the picture corresponding to the visual angle is directly shot; or a 3D object is placed in the graphical software, a camera position corresponding to the actual human eye position is configured, the actual human eye position is converted into the position of an observation space in the graphical software, and then the picture corresponding to the visual angle is generated.
The graphics software may be Opengl.
In step 4), since the cylindrical holographic directional diffusion screen is cylindrical, in order to display a correct image, it is necessary to convert a coordinate system and perform a pre-distortion process.
In step 5), the cylindrical holographic directional scattering screen does not scatter light horizontally incident to human eyes, and scatters light in a large range at other angles, so that a user can see a complete picture, and a seen 3D scene seems to be suspended in the center of the cylindrical system. As the user moves around the system, as if looking around the scene, the frame rate at which the image or dynamic video is displayed depends roughly on the frame rate at which the face is detected and the time of post-processing.
In step 5), since the DMD used by the projector is a 100% full offset projection, the projection range can only cover a half-circumference cylindrical surface after being reflected or transmitted by an aspheric mirror, and the viewing angle of the cylindrical screen is about 120 degrees due to the limitation of the scattering angle of the scattering screen. In order to enable the user to freely view the image in 360 degrees, a stepping motor is arranged below the projector, when the position of the face changes by a certain angle (such as from 0 degree to 30 degrees), the stepping motor is controlled to drive the projector to rotate, and a projection picture is changed simultaneously according to the rotating speed and angle of the stepping motor, so that the 3D view can be seamlessly connected during rotation without flickering.
In the step 5), the loudspeaker surrounds for 360 degrees, and voice interaction is provided on the basis of audio playing, so that more real experience is brought to people.
The transmission type projection imaging optical system or the reflection type projection imaging optical system in the 360-degree free three-dimensional display sound box based on the human face detection provided by the invention adopts an aspheric lens and an aspheric reflector to regulate and control light rays, and the transmitted or reflected light rays pass through an optical axis to be imaged to the opposite side. The 360-degree free three-dimensional display sound box based on the face detection provided by the invention realizes that the system meets the requirement of arbitrary viewing in the horizontal direction, also provides a viewing angle in the vertical direction, and is suitable for 1 to 2 people to view. The intelligent sound box system with voice interaction is realized by combining the display system and sound box equipment.
For the 360-degree free three-dimensional display sound box based on the face detection, provided by the invention, a user can watch a 3D scene in any horizontal direction and a certain vertical direction at a proper distance without wearing special glasses. The system has the advantages of easiness in use, low production cost, portability and mobility, and is suitable for application virtual exhibitions such as interactive sound boxes, museums and the like, teleconferences, multi-user online games and the like. This system is very promising for the low-cost portable 360-degree interactive market.
Drawings
Fig. 1 is a schematic structural diagram of a 360-degree free three-dimensional display sound box based on face detection;
FIG. 2 is a schematic view of a speaker configuration;
FIG. 3 is a schematic diagram of human eye position and virtual viewpoint position conversion;
FIG. 4 is a schematic diagram of a transmissive projection optical system;
FIG. 5 is a schematic diagram of a first cylindrical polarizer and a second cylindrical polarizer corresponding to the transmissive projection imaging optical system of FIG. 4;
FIG. 6 is a schematic diagram of a reflective projection optical system;
FIG. 7 is a detailed optical configuration of the transmission type projection imaging optical system of FIG. 4;
FIG. 8 is a detailed optical configuration of the reflective projection optical system of FIG. 6;
wherein, the human face detection module is used for detecting human faces at 1 and 360 degrees; 2. a cylindrical holographic directional scattering screen; 3. a projection imaging optical system; 4. a speaker; 5. a miniature projector module; 6. a first cylindrical polarizing plate; 7. a second cylindrical polarizing plate 2; 8. a stepper motor.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the following embodiments and accompanying drawings.
As shown in fig. 1, the three-dimensional display speaker includes a 360-degree face detection module 1, a micro projector module 5, a projection imaging optical system 3 (a transmission type projection imaging optical system or a reflection type projection imaging optical system), and a cylindrical holographic directional diffuser screen 2; the 360-degree face detection module 1 is arranged around the outer side of the three-dimensional display sound box, the cylindrical holographic directional scattering screen 2 is arranged in the center of the three-dimensional display sound box, and the projection imaging optical system 3 and the miniature projector module 5 are sequentially arranged below the cylindrical holographic directional scattering screen. As shown in fig. 2, a speaker 4 is located at the bottom of the system to play audio and provide a human experience of interacting with the 3D scene.
The cylindrical holographic directional scattering screen 2 is used as a display area and is positioned in the center of the system, and the cylindrical holographic directional scattering screen does not deflect light rays which are horizontally incident to human eyes, so that the screen is in a transparent state, and a scene behind the cylindrical screen can be seen. The cylindrical screen (cylindrical holographic directional scattering screen) performs large-angle scattering in other directions, so that a user can see a complete picture at different positions. The projection imaging optical system 3 is positioned at the upper part of the micro projector, projects the image projected by the micro projector onto the directional scattering screen, and completes the inverse transformation of the image.
Specifically, the 360-degree face detection module 1 includes a camera and a model Retinaface for detecting a face, and a face image acquired by the camera is input to the model Retinaface to output a face position. The camera is composed of four undistorted cameras with 90-degree visual angles in the horizontal direction and 60-degree visual angles in the vertical direction. As shown in fig. 3, since the face positions of the whole week need to be detected, a fast face detection frame rate is required, and for a panorama formed by splicing 4 images, the conventional face detection algorithm cannot meet the requirements. The human face detection model is based on Retinaface, the backbone network is MobileNet 2, the MobileNet 2 network is divided into stages 1, 2 and 3, the feature maps of the stages are sent into the FPN network structure to extract shallow, intermediate and deep information, the deep layer has a large receptive field, and can detect a large human face, the shallow layer positioning information is accurate, and the small human face can be detected. The dataset used for training is the WiderFace dataset. The trained model can meet the real-time requirement of the system. The model can accurately detect five landmark positions of eyes, nose and mouth of a human, and the average value of coordinates of two eyes is taken as the detected eye position. In order to increase the detection rate, the resolution of each camera is 480 × 360 smaller, and a 1920 × 360 panoramic face detection image is transversely spliced. Setting a horizontal total of 360 viewing angles and a vertical total of 60 viewing angles, allocating 5-6 pixels to each horizontal viewing angle and 6 pixels to each vertical viewing angle. For the case near 0 ° (360 °), the face detection model has better robustness, and can accurately detect even a part of the face is exposed, so that the case of no splicing at the edge is not considered. At the splicing position of two adjacent pictures, the model can accurately identify the face even if the pictures are slightly misaligned.
The human face detection module records the pixel value of the human eye position of each frame, so that the pixel range corresponding to the human eyes at each angle is obtained, and the pitch angle and the yaw angle of the human eyes relative to the camera position are calculated. Because there is some displacement between the camera position and the center of the cylindrical display, the vertical viewing angle that can be provided will be less than 60 degrees. When the face is too close to the camera, the camera cannot position the face, the 3D model cannot change the position according to the position of the eyes, and the correct view cannot be seen. It is calculated that the height of the diffuser screen (display area) is 125mm, and when the face is 215mm away from the system, the system can provide a vertical viewing angle of about 30 degrees. A suitable viewing distance should be greater than 215 mm. When the face distance is 310mm, the system can provide a vertical viewing angle of about 40 degrees. If the viewing angle is not required to be consistent with the real viewing angle, the vertical viewing angle range can be enlarged in the software according to equal proportion, namely, the user moves 1 degree in the vertical direction, and the camera movement 2 degrees is simulated in the software, so that the viewing range in the vertical direction is enlarged. If a real object is to be displayed, a camera can be used for shooting pictures corresponding to all the visual angles in advance, and the pictures corresponding to the visual angles are called in real time during display. If the virtual object is to be displayed, a camera position (observation space) corresponding to the actual eye position is configured in the graphics software (such as Opengl), and the actual eye position is converted to the position of the observation space in Opengl, so as to generate a picture corresponding to a visual angle. Since the screen is cylindrical, in order to display a correct image, the picture needs to be converted from a cartesian coordinate system to a polar coordinate system, and the image needs to be pre-distorted.
The present invention provides 2 kinds of projection imaging systems, respectively defined as a transmissive projection imaging optical system and a reflective projection imaging optical system, as shown in fig. 4 and 5, respectively.
As shown in fig. 4 and 7, the transmissive projection imaging optical system includes a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens, and an aspheric lens in sequence; the transmission type projection imaging optical system includes a first cylindrical polarizing plate 6 and a second cylindrical polarizing plate 7; the first cylindrical polaroid surrounds the aspheric lens, and the second cylindrical polaroid surrounds the cylindrical holographic directional scattering screen; the polarization directions of the first cylindrical polarizer and the second cylindrical polarizer are orthogonal to eliminate the influence of unscattered straight light.
Fig. 7 shows a specific optical structure of the transmissive system in fig. 4. The final lens group is more critical and consists of four aspheric surfaces. The light emitted by the micro projector module 5 reaches the last group of lenses through a plurality of lens groups, and finally reaches the image surface through aspheric surface transmission, aspheric surface reflection and aspheric surface transmission respectively. As shown in fig. 7, the light rays of the upper light ray that have undergone two reflections pass through the optical axis. The projected image is in polar form (circular ring shape), located on the outer ring (corresponding to the higher object height), projected through the system to the underside of the cylindrical screen, located on the inner ring (corresponding to the lower object height) projected above the cylindrical screen. Since it is directly transmitted from the lens, the influence of the direct transmission needs to be considered.
The light emitted by the micro projector module 5 passes through the transmission type projection imaging optical system, is obliquely incident to the first cylindrical polaroid 6 and then reaches the cylindrical holographic directional scattering screen 2, and the cylindrical holographic directional scattering screen 2 deflects the light and finally reaches human eyes through the second cylindrical polaroid 7.
In order to keep the transparent state of the cylindrical screen, the scattering screen does not deflect light rays which are horizontally incident to human eyes, namely, a scene behind the cylindrical screen can be seen, the cylindrical screen performs large-angle scattering in other directions so that a user can see a complete picture at different positions, and the seen 3D scene seems to be suspended in the center of the cylindrical system. The 360-degree face detection module 1 detects the position of a face (human eyes) in real time, generates a virtual viewpoint in software, so that a 3D scene picture of the corresponding viewpoint is taken, sent into a micro projector through a series of processing, scattered through a projection imaging light path and a scattering screen, and observed by human eyes. As the user moves around the system, as if looking around the scene, the frame rate at which the image or dynamic video is displayed depends on the frame rate at which the face is detected and the time of post-processing.
Fig. 5 shows the change of polarization state of light after passing through two polarizers with orthogonal polarization directions. Analyzing the light rays obliquely incident to the cylindrical polarizer 6, respectively establishing a polarizer coordinate system xyz and a light wave coordinate system x 'y' z ', setting the polarization direction of the polarizer 6 to be parallel to the x axis, the light transmission direction to be an xz plane, the light non-transmission direction to be a yz plane, and the vibration direction of the light to be perpendicular to the z' axis (i.e. the x 'y' plane), obtaining the intersecting line OA of the xz plane and the x 'y' plane as the light transmission direction, the intersecting line OB of the yz plane and the x 'y' plane as the light non-transmission direction, and the OA and OB are generally not 90 degrees. The light is scattered by the directional scattering screen 2, the polarization state of the scattered light is changed into natural light, but a part of the light is directly transmitted without being scattered, and bright spots can be seen if the human eyes directly observe the light. Therefore, a cylindrical polarizer 7 with the polarization direction perpendicular to the polarizer 6 is added on the periphery of the directional diffuser 2, and the light transmission direction of the polarizer 7 is the OB direction and the light non-transmission direction is the OA direction, so that the straight transmission light can be filtered out, and the scattered light can still be observed by people through the polarizer 7.
As shown in fig. 6 and 8, the reflective projection imaging optical system sequentially includes a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens, an aspheric lens, a sixth spherical lens, a seventh spherical lens, and an aspheric mirror; the second spherical lens and the third spherical lens form a double cemented lens. The last surface of the reflecting system is an aspheric reflecting mirror, so that the system does not need to consider the influence of straight light transmission and does not need to add two polarizing plates.
Fig. 8 shows a specific optical structure of the reflective projection imaging optical system in fig. 6. The upper side light emitted by the micro projector 5 module reaches the lower side of the aspheric surface reflector through the lens group, and is reflected by the aspheric surface reflector and then passes through the optical axis to be imaged on an image surface. Unlike the transmissive structure, the outer annular image of a higher object height projects to the upper side of the cylindrical screen and the inner annular image of a lower object height projects to the lower side of the cylindrical screen.
Because the DMD adopted by the projector is 100% full offset projection, the projection range can only cover a half-circumference cylindrical surface after being reflected or transmitted by an aspheric surface mirror, and the view field angle which can be observed by the cylindrical screen is about 120 degrees due to the limitation of the scattering angle of the scattering screen. In order to freely view in 360 degrees, a stepping motor 8 is arranged below the projector, when the position of a human face changes a certain angle (such as from 0 degree to 30 degrees), the stepping motor is controlled to drive the projector to rotate, and a projection picture is changed simultaneously according to the rotating speed and angle of the stepping motor, so that the 3D view can be seamlessly connected during rotation without flickering.
While the present invention has been described in further detail by way of illustration and example, it is to be understood that the invention is not limited to the precise embodiments and examples described above, and that the foregoing description is intended to be illustrative and not limiting, since various changes and modifications may be effected therein by one skilled in the art without departing from the scope or spirit of the invention as defined by the appended claims.
Claims (4)
1. A360-degree free three-dimensional display sound box based on face detection is characterized by comprising a 360-degree face detection module, a miniature projector module, a stepping motor transmission module, a transmission type projection imaging optical system or a reflection type projection imaging optical system, a cylindrical holographic scattering screen and a loudspeaker module; 360 degree face detection module arranges around the three-dimensional display audio amplifier outside, the periphery at the three-dimensional display audio amplifier is arranged to the holographic directional scattering screen of cylinder type, miniature projector module belongs to the ray apparatus submodule piece among the transmission-type or reflection-type projection imaging optical system, and on step motor drive module was arranged in to miniature projector module, step motor module arranged the system bottom in, avoided sheltering from the production of observation area, the system top is arranged in to the speaker module.
2. The 360-degree autostereoscopic three-dimensional display speaker according to claim 1, wherein the transmissive projection imaging optical system includes a micro projection module and a cylindrical polarizer; the miniature projection module sequentially comprises a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens and an aspheric lens; the cylindrical polarizer module comprises a first cylindrical polarizer and a second cylindrical polarizer; the first cylindrical polaroid surrounds the aspheric lens, the second cylindrical polaroid surrounds the cylindrical holographic scattering screen, and the polarization directions of the first cylindrical polaroid and the second cylindrical polaroid are orthogonal; the third spherical lens and the fourth spherical lens form a double cemented lens, and four surfaces of the aspheric lens are aspheric surfaces.
3. The 360-degree autostereoscopic three-dimensional display speaker according to claim 1, wherein the reflective projection imaging optical system includes a micro projection module, a large-caliber aspheric mirror; the miniature projection module sequentially comprises a first spherical lens, a second spherical lens, a third spherical lens, a fourth spherical lens, a fifth spherical lens, an aspheric lens, a sixth spherical lens and a seventh spherical lens; the second spherical lens and the third spherical lens form a double cemented lens.
4. The 360-degree autostereoscopic three-dimensional display speaker according to claim 1, wherein the 360-degree face detection module comprises a camera and a neural network model for detecting a face, and the model frame is based on Retinaface; the human face image collected by the camera is subjected to reasoning by the detection model, and the coordinates of the human eye position in the whole peripheral view field relative to the center of the display device are output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011212021.1A CN112351358B (en) | 2020-11-03 | 2020-11-03 | 360-degree free three-dimensional type three-dimensional display sound box based on face detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011212021.1A CN112351358B (en) | 2020-11-03 | 2020-11-03 | 360-degree free three-dimensional type three-dimensional display sound box based on face detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112351358A CN112351358A (en) | 2021-02-09 |
| CN112351358B true CN112351358B (en) | 2022-03-25 |
Family
ID=74355888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011212021.1A Active CN112351358B (en) | 2020-11-03 | 2020-11-03 | 360-degree free three-dimensional type three-dimensional display sound box based on face detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112351358B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115494961B (en) * | 2022-11-17 | 2023-03-24 | 南京熊大巨幕智能科技有限公司 | Novel interactive surrounding intelligent display equipment based on face recognition |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006339869A (en) * | 2005-05-31 | 2006-12-14 | Sanyo Electric Co Ltd | Apparatus for integrating video signal and voice signal |
| CN101630066A (en) * | 2009-08-17 | 2010-01-20 | 浙江大学 | Three-dimensional display device of full-view visual field based on high-speed projector |
| CN101950550A (en) * | 2010-09-28 | 2011-01-19 | 冠捷显示科技（厦门）有限公司 | Display device for displaying pictures at different angles based on visual angle of viewer |
| CN104410853A (en) * | 2014-10-20 | 2015-03-11 | 深圳市亿思达科技集团有限公司 | Three-dimensional holographic display system based on transmission |
| CN104506836A (en) * | 2014-11-28 | 2015-04-08 | 深圳市亿思达科技集团有限公司 | Personal holographic three-dimensional display method and device based on eyeball tracking |
| CN109031677A (en) * | 2018-08-03 | 2018-12-18 | 联想(北京)有限公司 | Electronic equipment and its projecting method |
| CN111325823A (en) * | 2020-02-05 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Method, device and equipment for acquiring face texture image and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101750868B (en) * | 2010-01-08 | 2011-07-27 | 浙江大学 | Panoramic space three-dimensional display device simultaneously having horizontal and pitching multiple visual fields |
| CN102298256B (en) * | 2011-08-24 | 2012-11-21 | 浙江大学 | Pitching multi-view-angle suspended 360-degree-view-field space three-dimensional display device |
-
2020
- 2020-11-03 CN CN202011212021.1A patent/CN112351358B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006339869A (en) * | 2005-05-31 | 2006-12-14 | Sanyo Electric Co Ltd | Apparatus for integrating video signal and voice signal |
| CN101630066A (en) * | 2009-08-17 | 2010-01-20 | 浙江大学 | Three-dimensional display device of full-view visual field based on high-speed projector |
| CN101950550A (en) * | 2010-09-28 | 2011-01-19 | 冠捷显示科技（厦门）有限公司 | Display device for displaying pictures at different angles based on visual angle of viewer |
| CN104410853A (en) * | 2014-10-20 | 2015-03-11 | 深圳市亿思达科技集团有限公司 | Three-dimensional holographic display system based on transmission |
| CN104506836A (en) * | 2014-11-28 | 2015-04-08 | 深圳市亿思达科技集团有限公司 | Personal holographic three-dimensional display method and device based on eyeball tracking |
| CN109031677A (en) * | 2018-08-03 | 2018-12-18 | 联想(北京)有限公司 | Electronic equipment and its projecting method |
| CN111325823A (en) * | 2020-02-05 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Method, device and equipment for acquiring face texture image and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于视场拼接的全视差三维显示系统;彭祎帆等;《光学子报》;20110930;第40卷(第9期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112351358A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10560687B2 (en) | LED-based integral imaging display system as well as its control method and device | |
| US6795241B1 (en) | Dynamic scalable full-parallax three-dimensional electronic display | |
| JP5358015B2 (en) | System and method for generating and displaying panoramic images and videos | |
| US7553023B2 (en) | Multi-dimensional imaging apparatus, methods, and systems | |
| CN101072366B (en) | Free stereo display system based on light field and binocular vision technology | |
| US20190007677A1 (en) | Systems and Methods for Convergent Angular Slice True-3D Display | |
| US9036003B2 (en) | Multi-pitching angle suspended 3D display device with 360-degree field of view | |
| US6813083B2 (en) | Device for reproducing three-dimensional image with background | |
| CN102123291B (en) | Intelligent naked-eye stereoscopic display system and control method thereof | |
| JP3744559B2 (en) | Stereo camera, stereo display, and stereo video system | |
| CN108919503B (en) | Integrated imaging 360-degree desktop 3D display system based on visual angle guide layer | |
| CN102692808A (en) | Large-scene 360-degree panorama dynamic display method, and display system | |
| Luo et al. | Pepper's Cone: An Inexpensive Do-It-Yourself 3D Display | |
| Gao et al. | 360 light field 3D display system based on a triplet lenses array and holographic functional screen | |
| CN102520527A (en) | Naked eye stereo display system and method | |
| CN112351358B (en) | 360-degree free three-dimensional type three-dimensional display sound box based on face detection | |
| US6178043B1 (en) | Multiview three-dimensional image display system | |
| RU2718777C2 (en) | Volumetric display | |
| JP2011197675A (en) | Projection system | |
| WO2003054625A1 (en) | A panoramic stereoscopic imaging method and apparatus | |
| JP2012008298A (en) | Three dimensional picture display device | |
| RU2526901C1 (en) | Three-dimensional display and method of forming three-dimensional images | |
| CN111818324A (en) | Device and method for generating three-dimensional large-visual-angle light field | |
| WO2000035204A1 (en) | Dynamically scalable full-parallax stereoscopic display | |
| CN104536254A (en) | Naked-eye three-dimensional display device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |