WO2020213555A1 - Dispositif de capture d'image, visiocasque, système d'affichage d'image, dispositif d'analyse d'image et procédé de sortie de données d'image - Google Patents

Dispositif de capture d'image, visiocasque, système d'affichage d'image, dispositif d'analyse d'image et procédé de sortie de données d'image Download PDF

Info

Publication number
WO2020213555A1
WO2020213555A1 PCT/JP2020/016258 JP2020016258W WO2020213555A1 WO 2020213555 A1 WO2020213555 A1 WO 2020213555A1 JP 2020016258 W JP2020016258 W JP 2020016258W WO 2020213555 A1 WO2020213555 A1 WO 2020213555A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
temperature
data
lens
lens parameter
Prior art date
Application number
PCT/JP2020/016258
Other languages
English (en)
Japanese (ja)
Inventor
村本　准一
Original Assignee
株式会社ソニー・インタラクティブエンタテインメント
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社ソニー・インタラクティブエンタテインメント filed Critical 株式会社ソニー・インタラクティブエンタテインメント
Publication of WO2020213555A1 publication Critical patent/WO2020213555A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/02—Mountings, adjusting means, or light-tight connections, for optical elements for lenses
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G02B7/30—Systems for automatic generation of focusing signals using parallactic triangle with a base line
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
    • G03B13/32—Means for focusing
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B19/00—Cameras
    • G03B19/02—Still-picture cameras
    • G03B19/04—Roll-film cameras
    • G03B19/07—Roll-film cameras having more than one objective
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/64—Constructional details of receivers, e.g. cabinets or dust covers

Definitions

  • the present invention relates to an image pickup device that captures an image used for display and analysis, a head mount display that displays based on the image, an image display system, an image analysis device that analyzes the image, and an image data output method by the image pickup device. ..
  • a game in which a user is photographed with a video camera, the image is replaced with another image, and the image is displayed on a display (see, for example, Patent Document 1).
  • a user interface system that receives mouth and hand movements taken by a video camera as application operation instructions.
  • technologies for acquiring the state of real space using captured images, reflecting them in displayed images, and performing some kind of information processing are available in a wide range of fields, from small mobile terminals to leisure facilities, regardless of their scale. It is used in.
  • a system that displays an image in a field of view corresponding to the line of sight of a user wearing a head-mounted display has also been developed.
  • a head-mounted display By using a head-mounted display, it is possible to enhance the immersive feeling in the image world and improve the operability of applications such as games.
  • a walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around in the space displayed as an image by physically moving.
  • parameters related to the camera such as focal length, optical center, and orientation of the optical axis have a great influence on the accuracy of the acquired information.
  • those values are acquired by prior calibration and treated as constants in the calculation used during operation to obtain the analysis result.
  • the analysis result will be affected, which in turn will lead to deterioration of the quality of the output data.
  • the camera is installed in a head-mounted display that is required to be lightweight, the problem is likely to become apparent because the limitation on the performance of the camera becomes large.
  • the present invention has been made in view of such a problem, and an object of the present invention is to provide a technique for performing analysis of a photographed image with stable accuracy regardless of the situation.
  • This imaging device includes a captured image acquisition unit that acquires data of an image captured by the imaging module, a temperature acquisition unit that acquires the temperature of the imaging module, and temperature characteristic data indicating changes in lens parameters with respect to temperature changes of the imaging module. It is provided with a lens parameter acquisition unit that acquires the value of the lens parameter with respect to the temperature acquired by the temperature acquisition unit, and an output unit that outputs the acquired lens parameter value in association with the image data. It is characterized by.
  • the head-mounted display is characterized by including the above-mentioned imaging device and a display panel for displaying an image reflecting the result of analyzing image data using the values of lens parameters.
  • Yet another aspect of the present invention relates to an image display system.
  • This image display system acquires lens parameter values and image data from the head mount display and the head mount display, analyzes the image data using the lens parameter values, and reflects the results. It is characterized by including an image generator that outputs an image to a head mount display.
  • Yet another aspect of the present invention relates to an image analyzer.
  • This image analysis device analyzes the captured image using the data acquisition unit that acquires the data of the captured image and the data corresponding to the value of the lens parameter depending on the temperature of the imaging module from the imaging device, and the value of the lens parameter. It is characterized by including an image analysis unit for performing an image analysis and an output data generation unit for generating output data based on the analysis result.
  • This image data output method refers to a step of acquiring image data captured by the imaging module, a step of acquiring the temperature of the imaging module, and temperature characteristic data indicating a change in lens parameters with respect to a temperature change of the imaging module. It is characterized by including a step of acquiring the value of the lens parameter with respect to the acquired temperature in the step of acquiring the temperature, and a step of associating the acquired lens parameter value with the image data and outputting the image data.
  • the captured image can be analyzed with stable accuracy regardless of the situation.
  • FIG. 1 It is a figure which shows the configuration example of the image analysis system to which this embodiment can be applied. It is a figure which shows the relationship between a focal length and an image height. It is a figure which shows the influence by temperature of the change of the image height with respect to the incident angle of light. It is a figure which shows the circuit structure of the image pickup apparatus in this embodiment. It is a figure which shows the internal circuit structure of the image analysis apparatus in this embodiment. It is a figure which shows the structure of the functional block of the signal processing apparatus and the image analysis apparatus in this embodiment. It is a figure which shows the example of the appearance shape of the head-mounted display to which the image pickup apparatus of this embodiment can be applied. It is a figure which shows the configuration example of the image display system including the head-mounted display of FIG.
  • FIG. 1 shows the basic configuration of an image analysis system to which this embodiment can be applied.
  • the image analysis system 1 includes an image pickup module 10, a signal processing device 12, and an image analysis device 16.
  • the image pickup module 10 includes at least a lens and an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).
  • the signal processing device 12 performs predetermined processing such as defect correction, noise removal, and demosaic processing on the image signal output from the image pickup module 10 to generate data of a captured image.
  • the image pickup module 10 and the signal processing device 12 may be realized as an image pickup device 14 including them integrally.
  • the image analysis device 16 acquires data of a captured image from the signal processing device 12, and acquires predetermined information by analyzing the data. For example, it detects a specific subject or acquires its position in a three-dimensional space.
  • the image analysis system 1 may be equipped with a stereo camera provided with two imaging modules 10. A technique is widely known in which images are taken from left and right viewpoints having a known interval by a stereo camera, and the distance to a subject is obtained by the principle of triangulation based on the difference in position (parallax) of corresponding feature points.
  • the image analysis device 16 may be a part of an image pickup device 14 including an image pickup module 10 and a signal processing device 12, or receives data of a captured image from the image pickup device 14 and performs a predetermined process using the data. It may be a part of an information processing device. In the latter case, the signal processing device 12 and the image analysis device 16 communicate with each other by wire or wirelessly. As described above, the combination of the image pickup module 10, the signal processing device 12, and the image analysis device 16 as an integrated device is not limited to the one shown in the drawing, and may be appropriately combined with another device having a function (not shown).
  • Parameters specific to the imaging device 14, which depend on the lens design and material of the imaging module 10, the mounting condition of the imaging module 10 in the imaging device 14, and the like, are acquired in advance by calibration.
  • the image analysis device 16 obtains an accurate analysis result by performing an analysis based on the analysis.
  • the refractive index of a lens changes depending on the temperature. Therefore, when the temperature inside the image pickup apparatus 14 rises due to heat generated by various circuits during operation, the focal length and the distortion reproduction coefficient of the lens change, which may have an effect that cannot be overlooked on the result of image analysis. This is particularly remarkable for resin lenses introduced in devices such as head-mounted displays and mobile terminals where weight reduction is desired.
  • ⁇ x Z ⁇ w / W
  • W is the number of pixels in the lateral direction of the camera
  • w is the visual field range in the lateral direction in the real space when the distance Z is 1, and depends on the viewing angle and the focal length.
  • FIG. 2 is a diagram showing the relationship between the focal length and the image height. Both (a) and (b) schematically show a state in which the image pickup module including the lens 8 is viewed from a bird's-eye view.
  • FIG. 3 shows the effect of temperature on the change in image height with respect to the incident angle of light.
  • the image height naturally increases, but the image height 18a at a temperature of 50 ° C. is slightly larger than the image height 18b at 25 ° C., and the difference becomes remarkable as the incident angle increases. ..
  • This characteristic is influenced not only by the focal length but also by the change in the distortion reproduction coefficient.
  • the difference 19 in image height is expressed by the number of pixels, a difference of about 20 pixels occurs when the incident angle is about 70 °. Such changes have an effect that cannot be overlooked in the above-mentioned calculation of distance.
  • the distortion of the image due to the lens is generally expressed by the following model formula.
  • x' (1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) x + 2p 1 xy + p 2 (r 2 + 2x 2 )
  • y' (1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) y + p 1 (r 2 + 2y 2 ) + 2p 2 xy
  • (x, y) is the position coordinate of the undistorted image in the coordinate system centered on the optical axis
  • r is the distance from the optical axis
  • (x', y') is the position coordinate of the distorted image. ..
  • k 1 , k 21 , and k 3 are coefficients representing radial strain
  • p 1 , p 2 are coefficients representing circumferential strain. In the present embodiment, these coefficients are referred to as strain reproduction coefficients. There is.
  • the shape of the image itself changes depending on its position on the image plane. That is, in addition to the distortion of the lens shape due to thermal expansion, the change in the refractive index of the lens also affects the change in the distortion of the image.
  • Such a change becomes a problem not only in the calculation of the distance to the subject described above but also in many image analyzes in which information is acquired based on the position of the feature point on the image. Therefore, in the present embodiment, the characteristics of the parameters that change with temperature, such as the focal length and the strain reproduction coefficient, are acquired in advance, and the true values of the parameters are acquired by actually measuring the temperature during operation and used for image analysis. ..
  • FIG. 4 shows the circuit configuration of the imaging device 14.
  • the image pickup apparatus 14 includes a temperature sensor 50, a CPU (Central Processing Unit) 42, a main memory 44, and a communication unit 46.
  • the temperature sensor 50 is a sensor such as a thermistor that continuously measures the temperature on the surface or inside of the image pickup module 10.
  • the CPU 42 is a main processor that processes and outputs signals such as image signals and sensor signals, as well as instructions and data.
  • the main memory 44 is composed of a RAM (Random Access Memory), and temporarily stores data, parameters, operation signals, etc. processed by the CPU 42.
  • the communication unit 46 includes a network adapter or an antenna, and transmits data input from the CPU 42 to the outside by wired or wireless communication. The communication unit 46 also receives data from the outside via wired or wireless communication via a network adapter or antenna and outputs the data to the CPU 42.
  • FIG. 5 shows the internal circuit configuration of the image analysis device 16.
  • the image analysis device 16 includes a CPU 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. Each of these parts is connected to each other via a bus 30.
  • An input / output interface 28 is further connected to the bus 30.
  • the input / output interface 28 outputs data to a peripheral device interface such as USB or IEEE1394, a communication unit 32 composed of a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or non-volatile memory, a display device, or the like.
  • An output unit 36, an input unit 38 for inputting data from an input device or the like, and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.
  • the CPU 23 controls the entire image analysis device 16 by executing the operating system stored in the storage unit 34.
  • the CPU 23 also executes various programs read from the removable recording medium, loaded into the main memory 26, or downloaded via the communication unit 32.
  • the GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to a drawing instruction from the CPU 23, and outputs the drawing process to the output unit 36.
  • the main memory 26 is composed of RAM and stores programs and data required for processing.
  • FIG. 6 shows the configuration of the functional blocks of the signal processing device 12 and the image analysis device 16 in the present embodiment.
  • the illustrated functional blocks can be realized by various circuits such as the CPU and main memory shown in FIGS. 4 and 5 in terms of hardware, and the data input function and data retention function loaded in the main memory in terms of software. It is realized by a program that exerts various functions such as various calculation functions. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various ways by hardware only, software only, or a combination thereof, and is not limited to any of them.
  • the signal processing device 12 includes a captured image acquisition unit 52 that acquires an image signal from the image pickup module 10, a temperature acquisition unit 56 that acquires the temperature of the image pickup module 10, and a lens parameter acquisition unit that acquires a predetermined parameter value according to the temperature. 58, a temperature characteristic storage unit 60 that stores temperature characteristics indicating the relationship between the temperature of the imaging module 10 and a predetermined parameter, a signal processing unit 54 that performs a predetermined process on the captured image, and data of the captured image and a predetermined parameter.
  • a data output unit 62 for associating and outputting values is provided.
  • the captured image acquisition unit 52 is realized by the CPU 42 or the like in FIG. 4, and acquires an electric signal of the captured image from the image sensor of the image pickup module 10.
  • the temperature acquisition unit 56 is realized by the CPU 42 or the like in FIG. 4, and continuously acquires the measured value of the temperature of the image pickup module 10 from the temperature sensor 50.
  • the lens parameter acquisition unit 58 acquires the value of the parameter at the temperature acquired by the temperature acquisition unit 56.
  • the parameters acquired here are typically the focal length and the distortion reproduction coefficient of the lens, but the types are not limited as long as they are parameters that change with temperature and affect the accuracy of image analysis.
  • the lens parameter acquisition unit 58 acquires the values of any one or two or more of them. Hereinafter, such parameters are collectively referred to as "lens parameters".
  • the temperature characteristic storage unit 60 is realized by the main memory 44 or the like in FIG. 4, and stores data representing the relationship between the temperature of the image pickup module 10 and the value of the lens parameter, which is acquired in advance by actual measurement or the like.
  • the data is in the form of a table showing lens parameters for discrete temperatures, for example.
  • the lens parameter acquisition unit 58 acquires the value of the lens parameter by appropriately interpolating the value represented by the table based on the measured value of the temperature.
  • the lens parameter acquisition unit 58 may acquire a value corresponding to the temperature at a predetermined time interval, or may acquire a value corresponding to the value at a timing when the temperature changes by a predetermined value or more. ..
  • the signal processing unit 54 is realized by the CPU 42, the main memory 44, etc. of FIG. 4, and performs signal processing such as defect correction, noise removal, and demosaic processing, which is also performed on a general captured image, to obtain data of the captured image. To generate. At this time, the signal processing unit 54 may also correct the lens distortion. For this correction, a model formula represented by the distortion reproduction coefficient of the lens described above can be used. Therefore, the signal processing unit 54 may acquire the distortion reproduction coefficient corresponding to the temperature at each time point acquired by the lens parameter acquisition unit 58, and perform correction using the distortion reproduction coefficient.
  • the data output unit 62 is realized by the CPU 42, the main memory 44, the communication unit 46, etc. of FIG. 4, and associates the captured image data generated by the signal processing unit 54 with the lens parameters acquired by the lens parameter acquisition unit 58. Output to the image analysis device 16. That is, the values of parameters used for image analysis in the image analysis device 16 such as the focal length and the distortion reproduction coefficient are output in association with the image data.
  • the data output unit 62 When the frame data of the captured image is sequentially output at a predetermined rate, the data output unit 62 outputs the frame data in association with the corresponding frame data only when the lens parameter acquisition unit 58 acquires a new value of the lens parameter. May be good.
  • the image analysis device 16 should output using the data acquisition unit 66 that acquires the data output by the signal processing device 12, the image analysis unit 68 that analyzes the captured image using the lens parameters of the data, and the analysis result.
  • An output data generation unit 70 for generating and outputting data is provided.
  • the data acquisition unit 66 is realized by the communication unit 32, the CPU 23, and the like in FIG. 5, and acquires data in which the captured image data and the lens parameter value are associated with each other from the signal processing device 12.
  • the image analysis unit 68 is realized by the CPU 23, GPU 24, main memory 26, etc. of FIG. 5, analyzes the captured image using the values of the lens parameters transmitted from the signal processing device 12, and acquires predetermined information.
  • the analysis means and the information to be acquired are not particularly limited.
  • the corresponding feature points may be extracted from the image captured by the stereo camera, and the distance of the subject may be obtained based on the parallax.
  • the above-mentioned parameter w is adjusted by using the focal length and the like acquired at the same time.
  • the accuracy of the calculated distance value can be maintained even if the temperature of the imaging module 10 changes.
  • by performing analysis using temperature-dependent lens parameters it is possible to derive results with stable accuracy for image shape and feature point-based analysis such as detection and recognition of objects.
  • the output data generation unit 70 is realized by the CPU 23, the output unit 36, etc. in FIG. 5, and generates and outputs output data using the result of image analysis. For example, a position in three dimensions is acquired using the distance of the subject, and an image of a virtual world corresponding to the position or movement is drawn. Alternatively, such a virtual object is superimposed on the captured image. If the image generated in this way is output to a display device (not shown), image representation such as virtual reality, augmented reality, and mixed reality can be realized.
  • the data output by the output data generation unit 70 is not limited to the display image, and the result of image analysis itself may be used as output data, or the analysis result is used as input information for information processing by itself and the result is output. May be good.
  • FIG. 7 shows an example of the appearance shape of the head-mounted display to which the image pickup apparatus of this embodiment can be applied.
  • the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104.
  • the mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it.
  • the wearing band 106 is made of a material or structure whose length can be adjusted according to the user's head circumference. For example, it may be an elastic body such as rubber, or a buckle or a gear may be used.
  • the output mechanism unit 102 includes a housing 108 having a shape that covers both eyes when the head-mounted display 100 is worn by the user, and includes a display panel inside so as to face the eyes when worn.
  • the display panel is realized by a liquid crystal panel, an organic EL panel, or the like.
  • the inside of the housing 108 is further provided with a pair of lenses located between the display panel and the user's eyes when the head-mounted display 100 is attached so that the image can be visually recognized in a wide field of view.
  • the head-mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ear when worn.
  • the head-mounted display 100 includes a camera 110 on the front surface of the output mechanism unit 102.
  • the camera 110 includes the imaging module 10 and the signal processing device 12 shown in FIG. 1, and photographs the real space at a predetermined frame rate in a field of view corresponding to the orientation of the user's face. Only one camera 110 may be provided on the front surface of the head-mounted display 100, or a stereo camera may be provided as shown in the figure. Further, the arrangement of the camera 110 is not particularly limited as long as it can photograph the front of the user.
  • the image taken by the camera 110 is subject to image analysis necessary for generating a display image, and can also be used as at least a part of the image displayed on the head-mounted display 100 itself. For example, if the captured image is used as a display image as it is, the user is in the same state as directly looking at the real space in front of him.
  • augmented reality and mixed reality can be realized by superimposing virtual objects at appropriate positions using the analysis results as described above.
  • FIG. 8 is a configuration example of an image display system including the head-mounted display of FIG. 7.
  • the head-mounted display 100 establishes communication with the image generator 200 by wire or wirelessly.
  • the image generator 200 may be further connected to the server via a network.
  • the server may provide the image generator 200 with an online application such as a game in which a plurality of users can participate via a network.
  • the image generation device 200 includes the image analysis device 16 of FIG. 1, sequentially receives data of images taken by the camera 110 of the head-mounted display 100, and acquires information related to the subject by image analysis. Then, the process of generating a display image reflecting the information and transmitting it to the head-mounted display 100 is repeated at a predetermined rate.
  • the image generation device 200 specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and displays the content display image so as to have a corresponding visual field. Generate at a predetermined rate.
  • the image analysis device 16 of the image generation device 200 acquires the position and posture of the head in the real space by a technique such as v-SLAM (Visual Simultaneous Localization And Mapping) using the captured image. Even in this case, by using the lens parameters that are accurate with respect to the temperature of the imaging module, it is possible to continue to accurately obtain the position and posture of the head.
  • the rotation angle and inclination of the head may be acquired by a motion sensor built in or externally attached to the head-mounted display 100, and may be used in a complementary manner with the analysis result of the captured image.
  • the head-mounted display 100 Since the head-mounted display 100 is worn by a person on the head, its weight reduction is a major issue. If the lens to be introduced into the camera 110 is made of resin, the weight can be reduced as compared with the lens made of glass, but the robustness to temperature is lowered, the quality of the display image using the analysis result is deteriorated, and in some cases, stereoscopic vision is performed. It is also possible that problems such as these may occur and cause poor physical condition. According to the present embodiment, stable analysis can be realized even with a resin lens, so that both image quality and weight reduction can be achieved at the same time.
  • FIG. 9 is a perspective view showing an example of the external shape of the image pickup module 10.
  • the image pickup module 10 is composed of a lens unit 80 having a built-in lens and a sensor unit 84 having a built-in image pickup element 82.
  • the lens unit 80 is composed of a lens barrel 86 capable of fixing a plurality of lenses in the optical axis direction, and a lens holder on the outer periphery thereof that holds the lens barrel 86 at a predetermined height from the sensor unit 84.
  • the lens barrel 86 is screwed into the inner wall of the lens holder 88, and the height of the lens can be adjusted by rotation during manufacturing or the like to keep the image sensor 82 in focus.
  • the lens holder 88 expands and deforms under the influence of heat.
  • FIG. 10 shows a cross-sectional view of the imaging module 10.
  • a lens holder 88a is joined to the upper surface of the sensor unit 84 incorporating the image sensor 82, whereby the lens barrel 86a is held.
  • the distance A between the principal point of the lens 8 and the surface of the image sensor 82 changes.
  • the distance A increases due to thermal expansion of the lens holder 88a as the temperature rises.
  • the number of lenses 8 is one in the figure, it is not intended to limit the number of lenses.
  • the range of the real space represented on the imaging surface changes according to the same principle as shown in FIG. 2, which affects the result of image analysis such as the distance to the subject.
  • the lens barrels 86a and 86b are held at the same height, but the heights of the screwed portions with the lens holders 88a and 88b are different. That is, the heights Ha and Hb of the lens holder 88a (a) and the lens holder 88b (b) where the threaded portion (for example, the threaded portion 89) is formed have a relationship of Ha ⁇ Hb.
  • the heights ha and hb where the threaded portion is formed in the lens barrel 86a of (a) and the lens barrel 86b of (b) are set to ha ⁇ hb.
  • the lens barrel holding mechanism by the lens holder is not limited to the screw, and in that case, the "screw part" can be rephrased as the "contact part" between the two or the "holding position” by the lens holder.
  • the contact portion may have a certain width as shown in the illustrated screw portion 89, and the height thereof is defined as the height of the contact portion such as the lower end, the upper end, or the midpoint of the contact portion from the bottom surface of the lens holder (the upper surface of the sensor portion 84). The distance to a predetermined position.
  • FIG. 11 shows the amount of change in the focal length of the lens and the position of the lens in the height direction with respect to temperature.
  • the white circles are the amount of change in the focal length
  • the dotted lines 94 and the solid line 96 are the amounts of changes in the lens position when the design heights of the contact portions of the lens holders are 4 mm and 8 mm, respectively.
  • the higher the contact portion is designed the larger the amount of change in the height of the lens with respect to the temperature change.
  • the contact height is designed to be 8 mm, the change in the lens position and the change in the focal length can be matched.
  • the optimum value of the height of the contact portion may exist depending on the rate of change of the focal length with respect to temperature ⁇ F / ⁇ t, the coefficient of linear expansion ⁇ of the lens holder, and the material and size of the lens and the lens holder. Recognize. Therefore, as shown in the figure, the height of the threaded portion so that the change in the lens position matches the change in the focal length is derived by an experiment or a theoretical formula after determining the material and size of the lens and lens holder to be introduced. To do. If the threaded portion of the lens holder and the lens barrel is formed according to the result, the change of the image on the captured image due to the temperature change can be suppressed, and the accuracy of the image analysis can be maintained.
  • the thermal expansion of the lens holder 88 also affects the external parameters of the image pickup device.
  • the optical axes of the lenses may not be parallel or the distance between the optical axes may change. Such a change affects the calculation of the distance of the subject described above.
  • the present embodiment by fixing the image pickup module in the vicinity of the principal point of the lens, the change of the external parameter with respect to the expansion of the lens holder is suppressed.
  • FIG. 12 shows the internal structure of the image pickup apparatus 14 including the substrate on which the image pickup module 10 is fixed.
  • the upper part of the figure shows a state in which the inside of the image pickup apparatus 14 is viewed from the direction of the lens.
  • a stereo camera in which two imaging modules 10a and 10b are arranged at a known interval is assumed, but the number of imaging modules 10 is not limited.
  • the imaging modules 10a and 10b are fixed by the substrate 90.
  • the substrate 90 may include at least a part of the circuit of the signal processing device 12 of FIG. 1 and is fixed to the housing of the image pickup device 14.
  • (A) and (b) are side views.
  • (A) is an internal structure of a general image pickup apparatus, and image pickup modules 10a and 10b are fixed on the upper surface of the substrate 90a.
  • the degree of freedom with respect to the position and orientation of the lens is relatively high, and it is conceivable that the distance and orientation of the optical axes of the left and right lenses change due to thermal expansion and deformation of the lens holder.
  • (B) shows the internal structure of the image pickup apparatus 14 of the present embodiment, and is fixed so that the principal points 92a and 92b of the lenses of the image pickup modules 10a and 10b and the upper surface of the substrate 90b coincide with each other.
  • the imaging modules 10a and 10b are penetrated through the substrate 90b, and are joined to the through holes of the substrate 90b on the side surfaces of the imaging modules 10a and 10b.
  • the vicinity of the principal points 92a and 92b of the lens is fixed, so that the distance and orientation of the optical axes of the left and right lenses are unlikely to change.
  • changes in external parameters can be reduced, and the influence of temperature changes on analysis accuracy can be suppressed.
  • the upper surface of the substrate 90b was aligned with the positions of the principal points 92a and 92b of the lens, but this embodiment is not intended to be limited to that.
  • the effect is exhibited only by fixing the side surfaces of the image pickup modules 10a and 10b with the substrate 90b.
  • joining to the substrate 90 within a predetermined range in the height direction from the principal points 92a and 92b of the lens it becomes more effective to maintain the analysis accuracy.
  • the change of the lens parameter with respect to the temperature change of the image pickup module is acquired in advance. Then, the temperature of the imaging module is measured during operation, and the values of the corresponding lens parameters are used to correct and analyze the captured image. As a result, no matter how the internal temperature changes due to long-term operation or the configuration of the device including the imaging module, the influence on the analysis result can be suppressed.
  • the processing accuracy is stable even in an external device that performs some processing using the captured image. Can be transformed into.
  • the influence of the increase in the focal length due to the temperature rise of the image pickup module can be appropriately offset by the influence of the thermal expansion of the lens holder. Since the structure can be optimized depending on the material, it is possible to deal with various material combinations, and in any case, the influence of temperature changes can be suppressed as a whole module.
  • the imaging module when assembling the imaging module as an imaging device, it is fixed to the substrate near the principal point of the lens. As a result, even if the lens holder expands or deforms due to heat, the principal point of the lens remains fixed to the image pickup apparatus, and the direction of the optical axis is less likely to change. Further, in the case of a multi-lens camera, the distance between the optical axes is less likely to change. As a result, image analysis can be performed with stable accuracy regardless of the temperature of the imaging module. Moreover, since the optical axis is fixed, stable display is possible even when the captured image is displayed as it is.
  • the above aspect is particularly effective for a resin lens that changes greatly with temperature. Therefore, if the imaging module or imaging device of the present embodiment is introduced into a head-mounted display, a mobile terminal, or the like, the weight of the device can be reduced and the quality of output data can be achieved at the same time.
  • the height of the lens holder 88 is such that the signal processing device 12 acquires the lens parameters corresponding to the temperature of the image pickup module 10 and uses them for correction or outputs them in association with the data of the captured image (FIG. 6). (FIG. 10) and the position where the image pickup module is fixed (FIG. 11) may be performed by any one of them or in combination of two or more. In any case, if the data related to the captured image is transmitted from the image pickup device 14 to the image analysis device 16, the robustness of the image analysis with respect to the temperature can be enhanced, and by extension, a high-quality image can be stably displayed.
  • the present invention can be used for various devices such as an image pickup device, a head-mounted display, an image analysis device, and a mobile terminal, and a system including the same.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Studio Devices (AREA)
  • Focusing (AREA)
  • Lens Barrels (AREA)
  • Cameras In General (AREA)
  • Automatic Focus Adjustment (AREA)

Abstract

L'invention concerne un dispositif de traitement de signal 12 d'un dispositif de capture d'image, dans lequel une unité d'acquisition d'image capturée 52 acquiert une image capturée par un module de capture d'image. Une unité d'acquisition de température 56 acquiert la température du module de capture d'image. Une unité d'acquisition de paramètre de lentille 58 se réfère à une caractéristique de température dans un paramètre de lentille, et acquiert une valeur de paramètre de lentille correspondant à la température acquise. Une unité de traitement de signal 54 utilise la valeur de paramètre de lentille acquise pour corriger l'image capturée. Une unité de sortie de données 62 associe le paramètre de lentille acquis et l'image capturée, et délivre celle-ci à un dispositif d'analyse d'image 16.
PCT/JP2020/016258 2019-04-19 2020-04-13 Dispositif de capture d'image, visiocasque, système d'affichage d'image, dispositif d'analyse d'image et procédé de sortie de données d'image WO2020213555A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-080090 2019-04-19
JP2019080090A JP2020178270A (ja) 2019-04-19 2019-04-19 撮像装置、ヘッドマウントディスプレイ、画像表示システム、画像解析装置、および画像データ出力方法

Publications (1)

Publication Number Publication Date
WO2020213555A1 true WO2020213555A1 (fr) 2020-10-22

Family

ID=72837451

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/016258 WO2020213555A1 (fr) 2019-04-19 2020-04-13 Dispositif de capture d'image, visiocasque, système d'affichage d'image, dispositif d'analyse d'image et procédé de sortie de données d'image

Country Status (2)

Country Link
JP (1) JP2020178270A (fr)
WO (1) WO2020213555A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002262156A (ja) * 2000-12-26 2002-09-13 Matsushita Electric Ind Co Ltd カメラ装置、カメラシステムおよび画像処理方法
JP2016052104A (ja) * 2014-09-02 2016-04-11 キヤノン株式会社 撮像装置、カメラシステム、画像処理装置および画像処理プログラム
WO2017126172A1 (fr) * 2016-01-18 2017-07-27 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002262156A (ja) * 2000-12-26 2002-09-13 Matsushita Electric Ind Co Ltd カメラ装置、カメラシステムおよび画像処理方法
JP2016052104A (ja) * 2014-09-02 2016-04-11 キヤノン株式会社 撮像装置、カメラシステム、画像処理装置および画像処理プログラム
WO2017126172A1 (fr) * 2016-01-18 2017-07-27 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement

Also Published As

Publication number Publication date
JP2020178270A (ja) 2020-10-29

Similar Documents

Publication Publication Date Title
JP7369147B2 (ja) 視認システムのホモグラフィ変換行列ベースの温度較正
US10269139B2 (en) Computer program, head-mounted display device, and calibration method
US20170324899A1 (en) Image pickup apparatus, head-mounted display apparatus, information processing system and information processing method
WO2017077918A1 (fr) Appareil de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations
JP6023801B2 (ja) シミュレーション装置
CN108535868B (zh) 头部佩戴型显示装置及其控制方法
Dansereau et al. A wide-field-of-view monocentric light field camera
CN114761909A (zh) 针对头戴式显示器的内容稳定
CN109309796A (zh) 使用多个相机获取图像的电子装置和用其处理图像的方法
JP2018055589A (ja) プログラム、物体の追跡方法、表示装置
CN211178345U (zh) 一种三维采集设备
CN109146961A (zh) 一种基于虚拟矩阵的3d测量及获取装置
JP2017224984A (ja) プログラム、装置、キャリブレーション方法
CN103517060A (zh) 一种终端设备的显示控制方法及装置
JP2019032599A (ja) 情報処理装置および位置情報取得方法
CN209279885U (zh) 图像采集设备、3d信息比对及配套物生成装置
JP6859447B2 (ja) 情報処理システムおよび対象物情報取得方法
TWI501193B (zh) Computer graphics using AR technology. Image processing systems and methods
CN113641088A (zh) 用于深度传感的时间数字转换器
CN109084679B (zh) 一种基于空间光调制器的3d测量及获取装置
JP7271215B2 (ja) 同期制御装置、同期制御方法、及びプログラム
US20170061695A1 (en) Wearable display apparatus, information processing apparatus, and control method therefor
JP6768933B2 (ja) 情報処理装置、情報処理システム、および画像処理方法
JP2020177163A (ja) 撮像装置およびヘッドマウントディスプレイ
WO2020213555A1 (fr) Dispositif de capture d'image, visiocasque, système d'affichage d'image, dispositif d'analyse d'image et procédé de sortie de données d'image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20790745

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20790745

Country of ref document: EP

Kind code of ref document: A1