US11483483B2 - Display apparatus - Google Patents

Display apparatus Download PDF

Info

Publication number
US11483483B2
US11483483B2 US17/298,027 US201817298027A US11483483B2 US 11483483 B2 US11483483 B2 US 11483483B2 US 201817298027 A US201817298027 A US 201817298027A US 11483483 B2 US11483483 B2 US 11483483B2
Authority
US
United States
Prior art keywords
display
display apparatus
distance
imaging
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US17/298,027
Other languages
English (en)
Other versions
US20220094857A1 (en
Inventor
Hiroshi Shimizu
Ikuya Arai
Yasunobu Hashimoto
Tomonori Nomura
Osamu Kawamae
Kazuhiko Yoshizawa
Naohisa TAKAMIZAWA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Maxell Ltd
Original Assignee
Maxell Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Maxell Ltd filed Critical Maxell Ltd
Assigned to MAXELL, LTD. reassignment MAXELL, LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ARAI, IKUYA, HASHIMOTO, YASUNOBU, KAWAMAE, OSAMU, NOMURA, TOMONORI, SHIMIZU, HIROSHI, TAKAMIZAWA, Naohisa, YOSHIZAWA, KAZUHIKO
Assigned to MAXELL HOLDINGS, LTD. reassignment MAXELL HOLDINGS, LTD. MERGER (SEE DOCUMENT FOR DETAILS). Assignors: MAXELL, LTD.
Assigned to MAXELL, LTD. reassignment MAXELL, LTD. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: MAXELL HOLDINGS, LTD.
Publication of US20220094857A1 publication Critical patent/US20220094857A1/en
Application granted granted Critical
Publication of US11483483B2 publication Critical patent/US11483483B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H04N5/232933—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
    • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
    • G06F1/1686—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N5/23229—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048—Indexing scheme relating to G06F3/048
    • G06F2203/04806—Zoom, i.e. interaction techniques or interactors for controlling the zooming operation

Definitions

  • the present invention relates to a display apparatus, particularly, to a technique effective for an image processing of displaying a state of superimposing a virtual object on a virtual reality image.
  • Patent Document 2 As another image display technique, known has been a technique in which an area where a photographer's field of view is interrupted on a display screen is imaged by a camera and is displayed on the display screen (see, for example, Patent Document 2), a technique in which an image display with a high degree of reality is realized by displaying an image according to a line-of-sight direction of a viewer who views the screen (see, for example, Patent Document 3), or the like.
  • PATENT DOCUMENT 1 Japanese Patent Application Laid-open No. 2006-238033
  • PATENT DOCUMENT 2 Japanese Patent Application Laid-open No. 2012-080193
  • PATENT DOCUMENT 3 Japanese Patent Application Laid-open No. 2012-190230
  • An object of the present invention is to provide a technique capable of providing a virtual space without uncomfortable feeling by displaying the virtual space so as to be equivalent to the real space.
  • a typical display apparatus has a first imaging unit, a first distance detector, a second imaging unit, a second distance detector, an image processor, and a display unit.
  • the first imaging unit is provided on a first surface and images a user.
  • the first distance detector identifies eyes of the user imaged by the first imaging unit and measures a first distance, which is a distance from the identified eyes to the first imaging unit.
  • the second imaging unit is provided on a second surface opposite to the first surface and images an object.
  • the second distance detector measures a second distance which is a distance from the second imaging unit to an object imaged by the second imaging unit.
  • the image processor performs an image processing on the images imaged by the first imaging unit and the second imaging unit.
  • the display unit displays the image image-processed by the image processor.
  • the image processor calculates a display imaging range displayed on the display unit based on the first distance measured by the first distance detector, the second distance measured by the second distance detector, and height of the display unit, extracts the calculated display imaging range from the image imaged by the second imaging unit, and causes the display unit to it.
  • the display imaging range calculated by the image processor is such a range that the object displayed on the display unit and an external scenery not interrupted by the display unit seen from a viewpoint of the user have the same size.
  • the image processor generates a virtual video with a display size adjusted according to the display imaging range and causes the display unit to display it.
  • the uncomfortable feeling about the image displayed on the display apparatus can be reduced.
  • FIG. 1 is an explanatory diagram showing an example of a configuration of a display apparatus according to a first embodiment
  • FIG. 2 is an explanatory diagram showing an example of control transition in the display apparatus of FIG. 1 ;
  • FIG. 3 is a schematic diagram showing an example of display adjustment by the display apparatus of FIG. 1 ;
  • FIG. 4 is explanatory diagram showing examples of an imaging range and a display range by an out-camera which the display apparatus of FIG. 1 has;
  • FIG. 5 is an explanatory diagram showing an example of correction of a display range according to an attachment position of the out-camera which the display apparatus of FIG. 1 has;
  • FIG. 6 is explanatory diagram showing an example of an image displayed on a display unit which the display apparatus of FIG. 1 has;
  • FIG. 7 is explanatory diagram showing an example of display in moving the display apparatus of FIG. 1 ;
  • FIG. 8 is explanatory diagram showing another example of FIG. 7 ;
  • FIG. 9 is explanatory diagram showing an example of an external appearance in the display apparatus of FIG. 1 ;
  • FIG. 10 is an explanatory diagram showing an example of use in a display apparatus according to a second embodiment.
  • FIG. 1 is an explanatory diagram showing an example of a configuration of a display apparatus 10 according to a first embodiment.
  • a display apparatus 10 includes a mobile terminal such as a smartphone or a tablet terminal. As shown in FIG. 1 , the display apparatus 10 is configured by a CPU (Central Processing Unit) 101 , a camera processor 102 , an image processor 103 , a memory unit 104 , a communication controller 105 , a sensor processor 106 , a display unit 107 , a voice input/output unit 108 , and the like.
  • a CPU Central Processing Unit
  • the CPU 101 controls each functional block by executing: an OS (Operating System) or application stored in the memory unit 104 ; a control program for controlling an operation of the display apparatus 10 ; or the like.
  • OS Operating System
  • application stored in the memory unit 104
  • control program for controlling an operation of the display apparatus 10 ; or the like.
  • the camera processor 102 is configured by an in-camera 1021 , an AF sensor 1022 , an out-camera 1023 , and an AF sensor 1024 .
  • the in-camera 1021 as a first imaging unit is a camera that is provided on a front surface side of the display apparatus 10 , that is, to a first surface that is on a display unit 107 side.
  • the AF sensor 1022 as a first distance detector measures a distance between the in-camera 1021 and a user of the display apparatus 10 to perform focus adjustment.
  • the out-camera 1023 as a second imaging unit is a camera provided on a side opposite to the in-camera 1021 , that is, on a back side of the display apparatus 10 which is a second surface.
  • the AF sensor 1024 as a second distance detector measures a distance between the out-camera 1023 and its imaging target to perform focus adjustment.
  • the sensor processor 106 is configured by a sensor group composed of a distance sensor 1061 , a gyro sensor 1062 , an acceleration sensor 1063 , and a magnetic sensor 1064 , etc.
  • the distance sensor 1061 is a sensor that monitors states such as a posture and a position of the display apparatus 10 .
  • the gyro sensor 1062 is a sensor that detects angular velocity of the display apparatus 10 in a rotation direction.
  • the acceleration sensor 1063 is a sensor that detects acceleration, and can detect gravity. The detection of gravity by the acceleration sensor 1063 makes it possible to detect an inclination etc. of the display apparatus 10 .
  • the magnetic sensor 1064 is a sensor that detects geomagnetism, and can detect an orientation (azimuth).
  • the image processor 103 is an image processing device called a so-called GPU (Graphics Processing Unit), and performs an image processing such as MR, VR, or AR.
  • the display unit 107 is configured by a display device and a drive controller for driving and controlling the display device, and displays an image image-processed by the image processor 103 and the like.
  • the display device is, for example, a liquid crystal display device, an organic EL (Electro Luminescence) device, or the like.
  • the memory unit 104 is composed of, for example, a semiconductor memory exemplified by a flash memory etc., and stores various kinds of pieces of information such as the above-mentioned application, OS, and control program, and setting information.
  • the applications stored in the memory unit 104 described above are AR applications, VR applications, MR applications, and the like that are applications for handling virtual objects and the like.
  • the communication controller 105 is a communication interface that wirelessly communicates with a router or the like via a wireless LAN (Local Area Network), and connects to an external network via the router or the like to send and receive information.
  • a wireless LAN Local Area Network
  • the communication controller 105 may be a communication interface that performs long-distance wireless communication with abase station of W-CDMA (Wideband Code Division Multiple Access), GSM (Global System for Mobile Communications) (registered trademark), or the like.
  • W-CDMA Wideband Code Division Multiple Access
  • GSM Global System for Mobile Communications
  • the voice input/output unit 108 includes a microphone for inputting external voice, a speaker for outputting voice outside, and the like.
  • FIG. 2 is an explanatory diagram showing an example of control transitions in the display apparatus 10 of FIG. 1 .
  • FIG. 3 is a schematic diagram showing an example of display adjustment by the display apparatus 10 of FIG. 1 .
  • a transition processing shown in FIG. 2 is performed when an AR application or the like stored in the memory unit 104 is activated.
  • a state J 207 which is a sensor processing
  • the image processor 103 takes in sensor information detected by the sensor processor 106 .
  • a distance (distance L 1 in FIG. 3 ) between the display apparatus 10 and the user who operates the display apparatus 10 and a distance (distance L 2 in FIG. 3 ) between the display apparatus 10 and an object 32 that the user gazes are measured.
  • This state J 21 is executed by the camera processor 102 .
  • performed respectively are a state J 201 that is an in-camera AF processing (AF processing for in-camera) and a state J 203 that is an in-camera processing, as well as a state J 202 that is an out-camera AF processing (AF processing for out-camera) and a state J 204 that is an out-camera processing.
  • a distance from the display apparatus 10 to user's eyes 31 is measured by using the in-camera 1021 and the AF sensor 1022 . Further, a distance from the display apparatus 10 to the object 32 is measured by using the out-camera 1023 and the AF sensor 1024 .
  • the AF sensor 1022 extracts distance information from the display apparatus 10 to the user's eyes 31 . Subsequently, in the state J 203 which is the in-camera processing, the AF sensor 1022 extracts a user's face imaged by the in-camera 1021 and recognizes the user's eyes 31 to extract eye position information (position information on the eyes).
  • the image processor 103 obtains distance information L 1 from the display apparatus 10 shown in FIG. 3 to the user's eyes 31 based on the above-mentioned distance information and eye position information.
  • the image processor 103 performs a processing such as cutout, enlargement, or reduction of a portion to be actually displayed by the display unit 107 from a range imaged by the out-camera 1023 based on the distance information L 1 acquired in the state J 203 , the distance information L 2 acquired in the state J 204 , and the sensor information fetched in in the state J 207 .
  • the distance L 2 which is a distance between the display apparatus 10 and the object 32 gazed by the user, may be infinity without the measurement by the AF sensor 1024 .
  • the display unit 107 displays an image generated by the image processor 103 in the state J 205 .
  • Height H 2 indicating height of a range to be displayed by the display apparatus 10 shown in FIG. 3 can be calculated (obtained) from the following formula by using the distance L 1 and distance L 2 described in the control transition of FIG. 2 , and height H 1 of the display screen of the display unit 107 in the display apparatus 10 .
  • H 2 ( L 1+ L 2)/ L 1 ⁇ H 1
  • a field angle (angle of view) el indicating an imaging range of the out-camera 1023 is set to be sufficiently wider than that captured by the user's eyes 31 , so that if an imaging range of the height H 2 at a position of the object 32 is extracted from a range imaged by the out-camera 1023 and is displayed by the display unit 107 , it can be displayed with a feeling of size for substantially the same size as that of surrounding scenery as seen from the user 1 .
  • the imaging range of the height H 2 which is to be a display imaging range, is trimmed from the range imaged by the out-camera 1023 and is displayed on the display unit 107 .
  • a lens provided in the out-camera 1023 is a lens having a wide field angle, that is, a so-called wide-angle lens.
  • the wide-angle lens is a fisheye lens, it can capture (take in) a scene in a range of approximately 180 degrees in vertical and horizontal directions. This makes it possible to photograph a front-side panoramic view at a wide angle.
  • the display apparatus 10 is configured by such a panel as to be itself a transparent glass plate, that is, by a so-called transparent liquid crystal display etc.
  • This transparent liquid crystal display is a display that transmits a background like glass by changing a polarizing film of the liquid crystal panel to a special film. By applying a predetermined signal to this transparent liquid crystal display, an image is displayed at an arbitrary position and with an arbitrary density.
  • the background can be seen through the display that transmits the background, so that only the virtual object needs to be displayed on the transparent liquid crystal display.
  • transparency to be set in advance may be set based on the density of the displayed image.
  • the virtual object can also be displayed at a coordinate position linked to an external image based on information on: detection of a position of the user's eye 31 by the in-camera 1021 ; an angle between the position and the panel; and the like.
  • FIG. 4 is an explanatory diagram showing examples of an imaging range and a display range by the out-camera 1023 that the display apparatus 10 of FIG. 1 has.
  • FIG. 4 shows a relationship between the imaging range of the out-camera 1023 and the display range obtained by the above-mentioned formula.
  • FIG. 4( a ) shows an example of cutting out and displaying a portion of a display range 42 from an imaging range 41 of the out-camera 1023 .
  • the imaging range 42 having Y 1 /M and X 1 /M multiplied by 1/M times in the vertical and horizontal directions as shown in FIG. 4( a ) is multiplied by M times and displayed on the display unit 107 .
  • FIG. 4( b ) shows an example in which an initial setting angle ⁇ 1 of the out-camera 1023 is multiplied by M times and is displayed by an enlargement processing according to a lens optical system of the out-camera 1023 so that it becomes a field angle ⁇ 2 .
  • FIG. 4 explained has been the example in which a line-of-sight head is directed to the center of the screen of the display apparatus 10 .
  • displaying a dominant eye out of both eyes as a target makes it possible to further create a state of viewing the real world.
  • the distance L 1 may be measured by using a dominant eye side as an origination during the distance measurement and be displayed and adjusted.
  • the setting of the dominant eye is made by software such as an MR application, a VR application, or an AR application, and setting information that has been set is stored in the memory unit 104 .
  • the CPU 101 recognizes the set dominant eye by reading the setting information stored in the memory unit 104 .
  • the setting of the dominant eye may be made, for example, when the software is started up, or the setting of the dominant eye may be made in an initial setting item(s) of the software. These settings are made from, for example, a menu etc. displayed on the display unit 107 .
  • FIG. 5 is an explanatory diagram showing an example of correction of a display range according to an attachment position of the out-camera 1023 that the display apparatus 10 of FIG. 1 has.
  • FIG. 5( a ) shows a relationship among the display apparatus 10 , a center point Ct of a display screen portion of the display unit 107 indicated by a dotted line, and an attachment position P of the out-camera 1023 . Further, FIG. 5( b ) shows a relationship between an imaging range 41 and a display range 42 of the out-camera 1023 .
  • the image processor 103 corrects a display range to be extracted from the imaging range 41 so as to become a display range 42 a that is respectively shifted by positions ⁇ X 1 and ⁇ Y 1 from a before-correction display range 42 b . This makes it possible to reduce an influence of a display deviation due to the attachment position of the out-camera 1023 .
  • the image which displays the object imaged by the out-camera 1023 can be seen so as to be on almost the same line as a user's eye line.
  • a display state by the correction shown in FIG. 5 will be described with reference to FIG. 6 .
  • FIG. 6 is an explanatory diagram showing an example of an image displayed on the display unit that the display apparatus 10 of FIG. 1 has.
  • an image 321 displayed on the display unit 107 is displayed at approximately the same size as that of an object seen from the user by performing the above-described size adjustment etc. to the object 32 in the real space hidden by the display apparatus 10 .
  • a virtual image 62 is a virtual object that is not imaged and displayed but is constructed by the software such as the VR application, MR application, or AR application described above.
  • This virtual image 62 can have consistency with the imaging image about size since the image processor 103 adjusts and draws its display size according to display magnification M.
  • the virtual image 62 can be arbitrarily arranged in a space of live video (picture).
  • the first embodiment has shown an example in which the virtual objects are superimposed on the imaged image, but a transmissive type display device can also be used as the display unit 107 as described above.
  • a transmissive type display device can also be used as the display unit 107 as described above.
  • the virtual object a virtual object to which the size adjustment has been performed is displayed so as to be viewed through the real space without displaying the imaged image.
  • a size of an imaging area in the real space can be calculated as described above, so that if a reference size of the virtual object is determined, the display size can be adjusted according to a display magnification ratio M.
  • FIG. 7 is an explanatory diagram showing an example of display in moving the display apparatus 10 of FIG. 1 .
  • FIG. 7( a ) shows an example of moving the display apparatus 10 horizontally or vertically.
  • ⁇ X shown in FIG. 7( a ) is an amount of movement of the display apparatus 10 in the horizontal direction
  • ⁇ Y is an amount of movement of the display apparatus 10 in the vertical direction.
  • the object 32 displayed on the display unit 107 changes from a state shown in FIG. 6 to display 321 shown in FIG. 7( a ) .
  • This is a state in which the display position has simply moved, so that adjustment can be performed by moving an extraction position of the imaged image with the out-camera 1023 in the same manner as described with reference to FIG. 5( b )
  • FIG. 7( b ) shows an example of being moved to ⁇ Z that is a screen-vertical direction.
  • ⁇ Z shown in FIG. 7( b ) is an amount of movement of the display apparatus 10 in the screen-vertical direction, that is, in front and back directions.
  • the magnification ratio after the movement can be calculated.
  • the movement amounts ⁇ X, ⁇ Y, and ⁇ Z can be measured by using the acceleration sensor 1063 etc. shown in FIG. 1 .
  • the display apparatus 10 performs the display adjustment again only when the CPU 101 determines that a measured value(s) of the acceleration sensor 1063 or the like exceeds a movement amount threshold that is the set movement amount. This makes it possible to suppress an increase in power consumption and a frequent change in the display screen.
  • a minute displacement of the display apparatus caused by the user holding it in a hand(s) can be suppressed by providing the camera processor 102 of FIG. 1 with a camera-shake preventing function not shown. This makes it possible to reduce difficulty in viewing the display screen displayed on the display unit 107 due to camera shake.
  • a distance from the display apparatus 10 to the user or the object 32 may be measured by using, for example, the distance sensor 1061 of FIG. 1 instead of using an autofocus function of the camera processor 102 .
  • used as the distance sensor 1061 is, for example, a sensor using infrared rays, ultrasonic waves, or the like.
  • the display apparatus 10 controls enlargement/reduction so that the virtual object is seen at almost the same appearance as that in the real space according to the magnification ratio M.
  • the appearance of the virtual object may be displayed slightly smaller or, conversely, slightly larger than that in the real space.
  • magnification setting item regarding display of enlargement, reduction, or equal magnification may be displayed on the display unit 107 so that the user can select desired display.
  • FIG. 8 is an explanatory diagram showing another example of FIG. 7 .
  • FIG. 8 shows a state in which the display apparatus 10 held by the hand of the user moves so as to be tilted from a vertical state and have a depression angle.
  • an imaging range of the out-camera 1023 is a range C to C′ as shown in the figure.
  • the imaging range of the out-camera 1023 becomes the range Cl to Cl′.
  • the image processor 103 maintains the image in a range interrupted by the display apparatus 10 from the user's eyes 31 and displays it on the display unit 107 .
  • the image in the range interrupted by the display apparatus 10 is slightly changed depending on an inclination of the display apparatus 10 , but becomes a range between the nearly ranges B to B′ shown by hatchings in FIG. 8 . Therefore, the image processor 103 causes the display unit 107 to display the image in the range between the nearly ranges B to B′ even if the imaging range changes due to the change in the inclination of the display apparatus 10 .
  • the display object viewed by the user on the display screen of the display unit 107 does not move with respect to the outer world of a portion not interrupted by the display apparatus 10 or its movement is reduced, so that the image with presence (realistic sensation) can be provided.
  • FIG. 9 is an explanatory diagram showing an example of an external appearance of the display apparatus 10 of FIG. 1 .
  • a switching button 2000 which is a software button, is displayed on the display unit 107 .
  • This switching button 2000 is a button for switching, for example, a display mode for displaying an image imaged by the out-camera 1023 (hereinafter, referred to as a camera display mode) and a display mode displayed by software such as an AR application for handling virtual objects (hereinafter, referred to as a transparent mode).
  • a display mode for displaying an image imaged by the out-camera 1023 hereinafter, referred to as a camera display mode
  • a display mode displayed by software such as an AR application for handling virtual objects
  • the camera display mode and the transparent mode are switched.
  • a switching processing is executed by the CPU 101 based on a signal inputted from the button 2000 , for example.
  • the display mode is the camera display mode or the transparent mode is displayed by a change of display color of the switching button 2000 , a change of a character(s), or the like, so that the user can easily understand which mode the display mode is.
  • a non-transparent-mode-compliant mark may be superimposedly displayed on an application icon.
  • FIGS. 3 to 8 shows an example in which the display apparatus 10 is held in a vertically long state, but the display apparatus 10 may be held in a horizontally long state.
  • the uncomfortable feelings felt by the user can be reduced by displaying the real world and the image displayed on the screen while the linkage therebetween is maintained so as to be seen at substantially the same size.
  • FIG. 10 is an explanatory diagram showing an example of use in a display apparatus 10 according to a second embodiment.
  • FIG. 10 shows an example in which the display apparatus 10 of FIG. 1 is used by a plurality of persons.
  • the display apparatus 10 is utilized by persons 82 and 83 .
  • the image processor 103 determines a person's face on a user side, which is imaged by the in-camera 1021 incorporated in the display apparatus 10 . At this time, the person 81 , the person 82 , and the person 83 are determined as persons. This makes it possible for the image processor 103 to recognize that the number of users is two or more.
  • the image processor 103 which has recognized that the number of users is two or more, determines the distance L 1 based on a result of face determination.
  • used as a method of determining the distance L 1 is, for example, one of the following first determination method, second determination method, third determination method, fourth determination method, or fifth determination method.
  • a first determination method is a method in which a distance to the person closest to the display apparatus 10 or to the person having the largest area imaged by the in-camera 1023 of the display apparatus 10 is set as the distance L 1 .
  • the CPU 101 identifies the person having the largest area among the persons to be imaged, acquires a distance to the identified person measured by the AF sensor 1022 as a distance L 1 , and outputs it to the image processor 103 .
  • a second determination method is a method in which a distance to the closest person to a center in an imaging range of the in-camera 1021 is set as the distance L 1 . Also in this case, the CPU 101 identifies the closest person to a central portion among the imaged persons, acquires a distance to the identified person measured by the AF sensor 1022 as the distance L 1 , and outputs it to the image processor 103 .
  • a third determination method is a method in which an average value of respective distances to the persons 80 , 81 , 82 imaged by the in-camera 1021 is set as the distance L 1 .
  • the CPU 101 acquires the distances to the persons 80 , 81 , 82 measured by the AF sensor 1022 , and calculates an average of the acquired distances. Then, the calculated distance is acquired as the distance L 1 and outputted to the image processor 103 .
  • a fourth determination method is a method in which a distance from the person holding the display apparatus 10 in a hand to the display apparatus 10 is set at the distance L 1 . From the shaking caused by the image processor 103 and the display apparatus 10 being held in the hand, a mutual relationship between analysis results of its shaking direction and shaking width and shaking of the imaged person(s) is taken, and the person who holds the display apparatus 10 in the hand is determined. Then, based on the determination result, the distance between the display apparatus 10 and the person's hand holding it is set at the distance L 1 .
  • a fifth determination method is a method of setting, as the distance L 1 , a distance from a user registered in advance to the display apparatus 10 .
  • a face of the user is photographed (captured) and registered in advance by the in-camera 1021 or the like.
  • a registration result thereof is stored in the memory unit 104 , for example.
  • the image processor 103 performs image recognition of the face captured by the image of the in-camera 1021 when the AR application or the like is activated, and determines that a face which has matched with the registered face is the user.
  • the CPU 101 may recognize a face first captured by the in-camera 1021 as a user and register the face.
  • the registered face has not been photographed for a preset time, the registered face is deleted and a face currently photographed may be registered as the user.
  • Which of the determination methods is selected may be made selectable by setting the display apparatus 10 , or may be made selectable by determining the determination method in advance and by storing, as setting information, the setting in the memory unit 104 when the AR application or the like is activated.
  • the CPU 101 measures a distance L 2 to an object 801 in the same manner as that in the example shown in FIG. 3 to obtain imaging height H 2 and an imaging field angle ⁇ 2 at that time, and calculates a display magnification ratio M. This makes it possible to display the image such as a virtual object on the display apparatus 10 in such feeling as to extract a part of the real world.
  • the display image of the display apparatus 10 viewed by a plurality of people and the real image of the visual field (eyesight) viewed by each person may differ due to some deviation in the line-of-sight direction.
  • the image processor 103 is set at a magnification ratio smaller than a value of the display magnification ratio M, and displays a range wider than an area hidden by the display apparatus 10 . This makes it possible to view the image in the line-of-sight direction also by any user who is viewing the display apparatus 10 . As a result, the usability of the display apparatus 10 can be improved.
  • the display magnification ratio M can be changed according to the number of persons recognized by the in-camera 1021 . If the number of imaged persons captured by the in-camera 1021 is one, the display magnification is set at M 1 ; if the number of imaged persons is two, the magnification ratio is set at M 2 ; and if the number of imaged persons is three, the magnification ratio is set at M 3 . Therefore, the display is adjusted according to the number of people viewing the display apparatus 10 , which improves usability.
  • constructing the virtual object in accordance with the above-described magnification ratio makes it possible to arrange the virtual object in an appropriate size on the display image.
  • the display apparatus 10 may construct an AR image by projecting it on a projection plate by a device such as a projector instead of a mobile terminal such as a smartphone or a tablet terminal.
  • the image to be displayed on the screen can be displayed while the linkage with the real world is maintained, so that the uncomfortable feeling which the users have can be reduced.
  • a part of the configuration of one embodiment may be replaced with the configuration of another embodiment, and the configuration of one embodiment may be added to the configuration of another embodiment.
  • another configuration may be added to a part of the configuration of each embodiment, and a part of the configuration of each embodiment may be eliminated or replaced with another configuration.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)
  • Closed-Circuit Television Systems (AREA)
  • User Interface Of Digital Computer (AREA)
  • Transforming Electric Information Into Light Information (AREA)
US17/298,027 2018-11-30 2018-11-30 Display apparatus Active US11483483B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2018/044149 WO2020110283A1 (ja) 2018-11-30 2018-11-30 表示装置

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/044149 A-371-Of-International WO2020110283A1 (ja) 2018-11-30 2018-11-30 表示装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/949,223 Continuation US11831976B2 (en) 2018-11-30 2022-09-21 Display apparatus

Publications (2)

Publication Number Publication Date
US20220094857A1 US20220094857A1 (en) 2022-03-24
US11483483B2 true US11483483B2 (en) 2022-10-25

Family

ID=70852225

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/298,027 Active US11483483B2 (en) 2018-11-30 2018-11-30 Display apparatus
US17/949,223 Active US11831976B2 (en) 2018-11-30 2022-09-21 Display apparatus

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/949,223 Active US11831976B2 (en) 2018-11-30 2022-09-21 Display apparatus

Country Status (4)

Country Link
US (2) US11483483B2 (ja)
JP (2) JP7146944B2 (ja)
CN (1) CN113170077A (ja)
WO (1) WO2020110283A1 (ja)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000276613A (ja) 1999-03-29 2000-10-06 Sony Corp 情報処理装置および情報処理方法
JP2006238033A (ja) 2005-02-24 2006-09-07 Matsushita Electric Ind Co Ltd 電子機器
JP2011248655A (ja) 2010-05-27 2011-12-08 Ntt Comware Corp ユーザ視点空間映像提示装置、ユーザ視点空間映像提示方法及びプログラム
JP2012080193A (ja) 2010-09-30 2012-04-19 Nec Casio Mobile Communications Ltd 携帯端末、カメラの倍率調整方法およびプログラム
JP2012190230A (ja) 2011-03-10 2012-10-04 Nec Casio Mobile Communications Ltd 表示装置、表示方法及びプログラム
US20130135295A1 (en) * 2011-11-29 2013-05-30 Institute For Information Industry Method and system for a augmented reality
JP2013258583A (ja) 2012-06-13 2013-12-26 Ntt Docomo Inc 撮像画像表示装置、撮像画像表示方法、プログラム
US20160078680A1 (en) 2014-09-17 2016-03-17 Dror Reif Technologies for adjusting a perspective of a captured image for display
US20180160093A1 (en) * 2016-12-05 2018-06-07 Sung-Yang Wu Portable device and operation method thereof

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011174993A (ja) * 2010-02-23 2011-09-08 Seiko Epson Corp 画像表示装置
GB201410285D0 (en) * 2014-06-10 2014-07-23 Appeartome Ltd Augmented reality apparatus and method
KR102489402B1 (ko) 2015-11-02 2023-01-18 삼성전자주식회사 디스플레이 장치 및 그의 영상 표시 방법

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000276613A (ja) 1999-03-29 2000-10-06 Sony Corp 情報処理装置および情報処理方法
JP2006238033A (ja) 2005-02-24 2006-09-07 Matsushita Electric Ind Co Ltd 電子機器
JP2011248655A (ja) 2010-05-27 2011-12-08 Ntt Comware Corp ユーザ視点空間映像提示装置、ユーザ視点空間映像提示方法及びプログラム
JP2012080193A (ja) 2010-09-30 2012-04-19 Nec Casio Mobile Communications Ltd 携帯端末、カメラの倍率調整方法およびプログラム
JP2012190230A (ja) 2011-03-10 2012-10-04 Nec Casio Mobile Communications Ltd 表示装置、表示方法及びプログラム
US20130135295A1 (en) * 2011-11-29 2013-05-30 Institute For Information Industry Method and system for a augmented reality
JP2013258583A (ja) 2012-06-13 2013-12-26 Ntt Docomo Inc 撮像画像表示装置、撮像画像表示方法、プログラム
US20160078680A1 (en) 2014-09-17 2016-03-17 Dror Reif Technologies for adjusting a perspective of a captured image for display
JP2017525052A (ja) 2014-09-17 2017-08-31 インテル コーポレイション 表示のためにキャプチャ画像の視野を調整する技術
US20180160093A1 (en) * 2016-12-05 2018-06-07 Sung-Yang Wu Portable device and operation method thereof

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion dated Feb. 12, 2019, received for PCT Application PCT/JP2018/044149, Filed on Nov. 30, 2018, 12 pages including English Translation.

Also Published As

Publication number Publication date
JP7146944B2 (ja) 2022-10-04
WO2020110283A1 (ja) 2020-06-04
JP2022183171A (ja) 2022-12-08
CN113170077A (zh) 2021-07-23
US20230017875A1 (en) 2023-01-19
US11831976B2 (en) 2023-11-28
JPWO2020110283A1 (ja) 2021-10-14
US20220094857A1 (en) 2022-03-24
JP7427739B2 (ja) 2024-02-05

Similar Documents

Publication Publication Date Title
KR102502404B1 (ko) 정보 처리 장치 및 방법, 그리고 프로그램
US11470377B2 (en) Display apparatus and remote operation control apparatus
US9282242B2 (en) Method and electric device for taking panoramic photograph
US8350896B2 (en) Terminal apparatus, display control method, and display control program
CN111034181A (zh) 图像捕获设备、图像显示系统和操作方法
US20150179147A1 (en) Trimming content for projection onto a target
JP2009064109A (ja) 画像投影装置及びその制御方法
JP6899875B2 (ja) 情報処理装置、映像表示システム、情報処理装置の制御方法、及びプログラム
JP2013258614A (ja) 画像生成装置および画像生成方法
KR101690646B1 (ko) 시 스루 디스플레이를 위한 카메라 구동 장치 및 그 방법
EP3076660B1 (en) Method and apparatus for displaying framing information
WO2015194084A1 (ja) 情報処理装置、情報処理システム、情報処理方法およびプログラム
JP2012095229A (ja) 画像表示装置及び画像表示装置用のコンピュータプログラム
WO2021238564A1 (zh) 显示设备及其畸变参数确定方法、装置、系统及存储介质
JP6792253B2 (ja) 画像表示装置、画像表示方法、及び画像表示プログラム
CN113870213A (zh) 图像显示方法、装置、存储介质以及电子设备
CN109417599B (zh) 便携终端
US11831976B2 (en) Display apparatus
CN113853569A (zh) 头戴式显示器
WO2019058641A1 (ja) 電子機器、プログラム、制御装置および制御方法
KR20190129982A (ko) 전자기기 및 그 제어 방법
US20240163391A1 (en) Information processing apparatus
JP7059829B2 (ja) 情報処理装置、情報処理方法及びプログラム
WO2022085127A1 (ja) 携帯端末装置
US20230209177A1 (en) Imaging apparatus

Legal Events

Date Code Title Description
AS Assignment

Owner name: MAXELL, LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SHIMIZU, HIROSHI;ARAI, IKUYA;HASHIMOTO, YASUNOBU;AND OTHERS;SIGNING DATES FROM 20210512 TO 20210520;REEL/FRAME:056379/0549

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: MAXELL HOLDINGS, LTD., JAPAN

Free format text: MERGER;ASSIGNOR:MAXELL, LTD.;REEL/FRAME:058255/0579

Effective date: 20211001

AS Assignment

Owner name: MAXELL, LTD., JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:MAXELL HOLDINGS, LTD.;REEL/FRAME:058666/0407

Effective date: 20211001

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE