WO2014080937A1 - 画像生成装置 - Google Patents
画像生成装置 Download PDFInfo
- Publication number
- WO2014080937A1 WO2014080937A1 PCT/JP2013/081272 JP2013081272W WO2014080937A1 WO 2014080937 A1 WO2014080937 A1 WO 2014080937A1 JP 2013081272 W JP2013081272 W JP 2013081272W WO 2014080937 A1 WO2014080937 A1 WO 2014080937A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- unit
- projection
- identification code
- light
- pattern
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/14—Measuring arrangements characterised by the use of optical techniques for measuring distance or clearance between spaced objects or spaced apertures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2513—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
- G01C3/02—Details
- G01C3/06—Use of electric means to obtain final indication
- G01C3/08—Use of electric radiation detectors
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
- G01C3/32—Measuring distances in line of sight; Optical rangefinders by focusing the object, e.g. on a ground glass screen
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
Definitions
- the present invention relates to an image generation apparatus that can acquire distance information to an object existing in an imaging space in association with a captured image.
- the illuminance detection means detects an illuminance greater than or equal to a predetermined value
- the state of the monitoring area imaged by the imaging means is displayed on the display device, and when the illuminance is less than the predetermined value, pattern light is emitted.
- An apparatus that displays obstacle information obtained by photographing reflected light and processing data on a display device is known (for example, see Patent Document 1).
- a system that performs optical ranging using a speckle pattern projects a primary speckle pattern from the lighting assembly into the target area, captures multiple reference images of the primary speckle pattern at different individual distances from the lighting assembly within the target area, and within the target area. Capture a test image of the primary speckle pattern projected on the surface of the object and identify the reference image that matches the primary speckle pattern most closely to the primary speckle pattern in the test image. Compared with the reference image, the position of the object is estimated based on the distance of the identified reference image from the lighting assembly (eg, US Pat.
- JP-A-6-87377 (second page, claim 1) JP-T 2009-528514 (paragraphs 0001, 0006, 0007) JP 2007-17643 (paragraphs 0003 and 0004)
- the conventional vehicle periphery monitoring device described above has a problem that it is necessary to switch the operation depending on the illuminance, and only one of the captured image and the distance information can be obtained.
- the above-described conventional optical distance measuring system has a problem that it is necessary to calculate a correlation value by pattern matching between a photographed pattern and a plurality of reference patterns, and the amount of calculation is large.
- the present invention has been made in order to solve the above-described problems, and enables to acquire both a captured image and distance information even in a bright illuminance environment, and projection of each spot in a captured image with a small amount of calculation. It is an object to specify a position in a pattern and generate an image with distance information.
- An image generation apparatus includes: A projection unit that projects pattern light of a predetermined wavelength into the imaging space; An imaging unit for imaging the imaging space; A separation unit that separates pattern projection image components by obtaining a difference between an imaging signal obtained when pattern light is projected and an imaging signal obtained when pattern light is not projected, among imaging signals obtained by imaging by the imaging unit; , A distance information generation unit that generates distance information based on the projection image component separated by the separation unit; The distance information generator is picked up from the arrangement of light spots in the projection image represented by the projection image component and the relationship between the position of each light spot in the projection pattern and the projection angle stored in advance.
- the pattern light projected from the projection unit includes a plurality of cells in an on state or an off state, which constitute an identification code accompanying each light spot
- the distance information generation unit specifies the position in the projection pattern of the light spot accompanied by the identification code based on the identification code
- the identification code associated with each light spot is composed of a plurality of cells arranged in the first direction in the projection pattern, and is arranged on the one side in the second direction in the projection pattern with respect to the light spot.
- the identification code is There is no more than one change point where the cell constituting the second part of the identification code changes from the on state to the off state or from the off state to the on state between adjacent light spots in the first direction. , The first portion of the identification code is the same between adjacent light spots in the second direction.
- the image generation apparatus includes: A projection unit that projects pattern light of a predetermined wavelength into the imaging space; An imaging unit for imaging the imaging space; A separation unit that separates pattern projection image components by obtaining a difference between an imaging signal obtained when pattern light is projected and an imaging signal obtained when pattern light is not projected, among imaging signals obtained by imaging by the imaging unit; , A distance information generation unit that generates distance information based on the projection image component separated by the separation unit; The distance information generator is picked up from the arrangement of light spots in the projection image represented by the projection image component and the relationship between the position of each light spot in the projection pattern and the projection angle stored in advance.
- the pattern light projected from the projection unit includes a plurality of cells in an on state or an off state, which constitute an identification code accompanying each light spot,
- the distance information generation unit specifies the position in the projection pattern of the light spot accompanied by the identification code based on the identification code,
- the identification code has only one change point where the cell constituting the identification code is switched from the on state to the off state or from the off state to the on state between adjacent light spots in the first direction in the projection pattern. It is determined to be.
- both a captured image and distance information can be acquired even in a bright illuminance environment, and distance information associated with the image can be obtained. Further, the position in the projected pattern for each light spot in the captured image can be specified, and distance information to the subject can be acquired with a small amount of calculation.
- FIG. 2 is a layout diagram of an imaging unit and a projection unit according to Embodiment 1 of the present invention. It is the schematic which shows the structure of the projection part 22 of FIG. It is a block diagram which shows the structural example of the isolation
- (A) And (b) is a figure which shows the ratio of the magnitude
- (A)-(c) is a figure which shows the arrangement
- FIG. 1 is a block diagram showing a configuration of an image generation apparatus according to Embodiment 1 of the present invention.
- the illustrated image generation apparatus includes an image acquisition unit 10, a pattern light generation unit 20, and a control unit 30.
- the image acquisition unit 10 includes an imaging unit 11, and the pattern light generation unit 20 includes a projection unit 22.
- FIG. 2 three-dimensionally represents the imaging space (imaging target space) JS together with the projection unit 22 and the imaging unit 11.
- FIG. 2 it is assumed that there is a rectangular parallelepiped subject OJ1 and a spherical subject OJ2 in the imaging space JS.
- the imaging unit 11 receives light from the subjects OJ1 and OJ2 in the imaging space JS and performs imaging.
- the image generation apparatus of the present invention obtains image information and distance information for each part of the image by obtaining distances to the respective parts of the captured subjects OJ1 and OJ2 based on information obtained by imaging.
- the projection unit 22 projects pattern light that generates a projection pattern toward the imaging space JS as shown in FIG.
- the projection pattern forms light spots arranged in a matrix, that is, in a horizontal direction (row direction) and a vertical direction (column direction).
- the “direction” relating to the projection pattern means a direction in the projection pattern formed when the pattern light is projected onto a virtual plane perpendicular to the optical axis, unless otherwise described.
- array and “position” in the projection pattern in the following description.
- FIG. 3 is a view of the imaging unit 11 and the projection unit 22 and one light spot SP formed at an arbitrary point on the subjects OJ1 and OJ2 in the imaging space as viewed from above.
- the imaging unit 11 and the projection unit 22 are disposed apart from each other by a distance Lpc in the horizontal direction. That is, the imaging unit 11 and the projection unit 22 are arranged at different positions in the horizontal direction, and are arranged at the same position in the vertical direction (up and down direction).
- a straight line connecting the imaging unit 11 and the projection unit 22 is referred to as a base line BL, and the distance Lpc is referred to as a base line length.
- the horizontal direction in the projection pattern corresponds to the direction of the base line BL, that is, the horizontal direction, and the vertical direction corresponds to a direction orthogonal to the horizontal direction.
- a light spot SP is formed on one of subjects OJ1 and OJ2 in the imaging space JS by the light projected from the projection unit 22, and the light from the light spot SP is received by the imaging unit 11.
- the projection angle ⁇ from the projection unit 22 to the light spot SP, the incident angle ⁇ from the light spot SP to the imaging unit 11, and the baseline length Lpc are known, based on the principle of triangulation, the subject from the baseline BL
- the distance Z to the light spot SP on OJ1 and OJ2 can be obtained by calculation.
- the projection angle ⁇ is an angle formed by a line perpendicular to the base line BL and a line connecting the projection unit 22 and the light spot SP in a plane including the base line BL and the light spot SP.
- the incident angle ⁇ is an angle formed by a line perpendicular to the base line BL and a line connecting the imaging unit 11 and the light spot SP in a plane including the base line BL and the light spot SP as shown in FIG.
- the incident angle ⁇ in the imaging unit 11 can be obtained based on the position on the imaging surface of the imaging unit 11 where the image of the light spot SP is formed, the direction of the axis of the imaging element, and the angle of view.
- the projection angle ⁇ from the projection unit 22 is determined in advance by the configuration of the projection unit 22 and is therefore known.
- the light spots on the imaging surface are known if the respective projection angles are known.
- the projection angle of each light spot is estimated based on the mutual relationship between the positions on the image. in this case, (A) “The magnitude relationship between the projection angles of the respective light spots in the projection unit 22 (for example, the order when the light spots are arranged in ascending order) and the magnitude relationship between the incident angles of the respective light spots in the imaging unit 11 (from the smallest order) Are the same order) Is satisfied, and based on this, the projection angle of each light spot imaged by the imaging unit 11 is known.
- each projection angle of the light spot imaged by the imaging unit 11 can be calculated with a small amount of calculation. It is possible to estimate accurately.
- condition (a) above is always satisfied with respect to the magnitude relationship of the angles in the vertical direction for the light spots projected at different angles in the plane perpendicular to the base line BL. There is no need to consider “replacement”.
- the base line BL extends in the horizontal direction and the light spots of the pattern light are arranged in the horizontal direction and the vertical direction will be further described.
- the pattern light generation unit 20 includes a drive unit 21 in addition to the projection unit 22, and the projection unit 22 includes a laser light source 23, a collimator lens, as shown in FIGS. 24, an aperture 25, and a diffraction grating 26.
- the drive unit 21 is controlled by the control unit 30 to cause the laser light source 23 to emit light, and the laser light emitted from the laser light source 23 is collimated by the collimator lens 24 and is made a predetermined beam diameter by the aperture 25.
- the diffraction grating 26 projects pattern light for generating a predetermined projection pattern onto the imaging space JS.
- the imaging unit 11 includes a lens 12 and an imaging element 14, and the image acquisition unit 10 includes an A / D conversion unit 15, a separation unit 16, in addition to the imaging unit 11, An image generation unit 17, a distance information generation unit 18, and a display processing unit 19 are provided.
- the lens 12 focuses the subject image on the imaging surface of the image sensor 14.
- the imaging element 14 outputs an imaging signal obtained by photoelectrically converting the incident image.
- the imaging element 14 has R, G, and B pixels arranged in a Bayer shape, and R, G, and B signals are output as imaging signals.
- the imaging unit 11 including the lens 12 and the imaging element 14 captures the subjects OJ1 and OJ2 in the imaging space JS. This imaging is performed at a predetermined frame frequency, and images of a plurality of continuous frames are obtained by imaging.
- the A / D conversion unit 15 converts the output of the imaging unit 11 into, for example, an 8-bit (256 gradation) digital signal D15.
- the separator 16 receives the output of the A / D converter 15, that is, the A / D converted image signal D15, and separates the projection image component and the background component.
- the image generation unit 17 generates a background image from the background components output from the separation unit 16.
- the distance information generation unit 18 generates distance information from the projection image component output from the separation unit 16.
- the display processing unit 19 displays the distance information generated by the distance information generation unit 18 in association with the background image generated by the image generation unit 17.
- An image (a signal indicating the distance information) associated with the distance information output from the display processing unit 19 is output to a display device (not shown) or the like.
- the control unit 30 controls the pattern light generation unit 20 and the image acquisition unit 10. For example, the control unit 30 controls the imaging mode, the frame rate, the exposure time, and the like for the imaging device 14 of the imaging unit 11 and sets the display mode, the display mode for distance information, and the like for the display processing unit 19. .
- the control unit 30 also supplies a signal for controlling the operation timing to the A / D conversion unit 15.
- the control unit 30 further sets an operation mode for the pattern light generation unit 20 and the image acquisition unit 10.
- the control unit 30 further includes a relationship between an identification code (described later) associated with each light spot included in the projection pattern projected from the projection unit 22 and a position of the light spot associated with the identification code in the projection pattern.
- Information Sdp, information Spa indicating the correspondence between the position on the projection pattern and the projection angle, information Szv indicating the axial direction and angle of view of the imaging unit 11, and information indicating the base line length Lpc are stored. The information is supplied to the information generator 18.
- the control unit 30 also performs control for synchronizing the operation of the pattern light generation unit 20 and the operation of the image acquisition unit 10. More specifically, the control unit 30 controls the imaging unit 11 to repeat imaging at a predetermined frame frequency, and causes the laser light source 23 to be in a light emitting state or a non-light emitting state every other frame. Further, the driving unit 21 is controlled, and a signal Snf indicating whether the laser light source 23 is in a light emitting state or a non-light emitting state is supplied to the separating unit 16.
- the frame frequency of the imaging unit 11 is, for example, 30 fps, and one frame image (a signal representing) D11 is output from the imaging unit 11 for each frame period.
- the timing of imaging each frame is controlled by the control unit 30.
- the projection unit 22 Since the laser light source 23 of the projection unit 22 alternately emits light and does not emit light every other frame, the projection unit 22 has a state in which pattern light is projected onto the imaging space JS and a state in which pattern light is not projected.
- the imaging unit 11 alternately obtains an image when the pattern light is projected and an image when the pattern light is not projected alternately every other frame.
- the separation unit 16 Based on the image when the pattern light is projected and the image when the pattern light is not projected, the separation unit 16 removes the pattern light image (projected image component) and the image from which the pattern light component has been removed ( Background component). That is, an image obtained in a frame period in which pattern light is not projected is output as a background component, and pattern light is derived from an image obtained in a frame period in which pattern light is projected, of two successive frame periods. An image obtained by subtracting an image obtained in a frame period in which is not projected is output as a projection image component.
- FIG. 5 is a block diagram illustrating a configuration example of the separation unit 16.
- the output (digital imaging signal) D15 of the A / D converter 15 is supplied to the input terminal 60.
- the frame delay unit 61 delays the digital imaging signal D15 supplied to the input terminal 60 by one frame period and outputs a frame delayed imaging signal D61.
- the difference calculation unit 62 is a difference between the imaging signal D15 and the frame delay imaging signal D61 (a difference obtained by subtracting imaging of a frame on which pattern light is not projected from an imaging signal of a frame on which pattern light is projected). And the difference signal D62 is output.
- the switch 63 closes at the timing when the imaging signal D15 for the frame on which the projection unit 22 does not project pattern light is supplied to the input terminal 60, and generates the image as the background component D63 via the output terminal 65. Supply to unit 17.
- FIGS. 6A to 6D show an example of the operation of the separation unit 16.
- the pattern light is not projected in the first frame PS1 and the third frame PS3, and the pattern light is not projected in the second frame PS2 and the fourth frame PS4. It is supposed to be projected.
- a captured image as shown in FIG. 6B is obtained in each frame period.
- the switch 63 is closed, and the imaging signal D15 at that time (the imaging signal D15 in the first frame, that is, the imaging unit output D11 in the state where the pattern light is not projected is converted into a digital signal.
- the signal D15) thus obtained is supplied as the background component D63 from the output terminal 65 to the image generation unit 17 (FIG. 6D).
- the imaging signal D15 is input to the frame delay unit 61.
- the difference calculation unit 62 digitally converts the imaging signal D15 at that time (the imaging signal D15 in the second frame, that is, the imaging unit output D11 in a state where the pattern light is projected).
- the output D61 (imaging signal of the first frame PS1) of the delay unit 61 is subtracted from the signal D15 obtained in this way, and a subtraction result (difference) D62 is output (FIG. 6C).
- the switch 63 is closed, and the imaging signal D15 at that time is supplied from the output terminal 65 to the image generation unit 17 as the background component D63.
- the imaging signal D15 is input to the frame delay unit 61.
- the difference calculation unit 62 subtracts the output D61 of the frame delay unit 61 from the imaging signal D15 at that time, and outputs a subtraction result (difference) D62. Thereafter, the same processing is repeated, and an image of only the background component and an image of only the projection image component are output every other frame period.
- the output of the image sensor 14 is, for example, an image signal in which R pixel values, G pixel values, and B pixel values are arranged in a Bayer array, and the output of the A / D converter 15 is also a digital signal corresponding thereto.
- 6A to 6D show images in which pixel values of all the pixels are obtained by interpolation for convenience of explanation.
- the separation processing by the frame delay unit 61, the difference calculation unit 62, and the switch 64 of the separation unit 16 is performed separately for each of R, G, and B, and the projection image obtained as a result of the separation is processed.
- the image generation unit 17 includes a video signal processing unit 72, and performs color interpolation processing (at Bayer arrangement at each pixel position) on an image composed of background components output from the separation unit 16. Interpolation of missing color components), gradation correction processing, noise reduction processing, contour correction processing, white balance adjustment processing, signal amplitude adjustment processing, color correction processing, etc., and the image obtained as a result of these processing Output as background image.
- the distance information generation unit 18 is information representing the distance from the imaging unit 11 to each part of the projection image based on the projection image component output from the separation unit 16 and the information about the projection pattern separately supplied from the control unit 30. Is generated.
- a projection pattern including an identification code in addition to a light spot is used. Therefore, before describing the operation of the distance information generation unit 18, the projection pattern used in the present embodiment will be described.
- the projection image (projection pattern) projected by the projection unit 22 includes light spots arranged in a matrix as shown in FIG. It has a group of dots that act as codes.
- FIG. 8 shows an enlarged part of the projection pattern.
- the minimum cell is called a dot position or a cell, and is a minimum unit that can be controlled to be on (a state where light is irradiated) or off (a state where light is not irradiated) in the projection pattern.
- cells of 480 rows in the vertical direction and 650 columns in the horizontal direction are formed in the projection range.
- a dot is constituted by a cell irradiated with light.
- Each light spot MK is formed so as to occupy an area composed of cells in the on state in two rows in the vertical direction and two columns in the horizontal direction.
- the light spot is also called a position confirmation marker or simply a marker.
- Light spots and dots may be collectively referred to as projection points.
- Each of the upper and lower rows and the left and right columns around the area of 2 rows and 2 columns is an area composed of off-state cells (cells not irradiated with light), and 4 rows and 4 including this area and the 2 rows and 2 columns area.
- the row area is called a spot area MA.
- DCa is a region that is adjacent to the right side of the spot region MA and is arranged in a row (a group of four dot positions that are adjacent to the spot region MA on the right side and aligned with each other). This is a region constituting the partial DCb.
- the four cells of the first part DCa are indicated by reference numerals c1 to c4, respectively, and the four cells of the second part DCb are indicated by reference numerals c5 to c8, respectively.
- Each cell of the first portion DCa and the second portion DCb can take either an on state (irradiated state) or an off state (non-irradiated state), and by this combination of on and off, An 8-bit identification code DC is configured.
- the identification code DC associated with each light spot MK is used for identifying the light spot MK.
- the cell adjacent to the right side of the first part DCa that is, the cell cbr adjacent to the lower side of the second part DCb is in the off state.
- the entire projection pattern is configured by repeating a region MB composed of cells of 5 rows and 5 columns, in which the identification code DC and the cell cbr are added to the spot region MA of 4 rows and 4 columns.
- the light spot MK is used to specify the position of each part of the projection pattern, and is composed of 2 rows and 2 columns of dots. Therefore, the light spot MK has a relatively large area in the imaging unit 11, and thus appears as a relatively high-luminance part.
- the identification code DC associated with each light spot MK is used to determine which of the many light spots included in the projection pattern is the light spot MK.
- FIG. 9 shows an example of an identification code used in the projection pattern.
- different “values” up to 55 ie different on / off combinations of identification codes are used.
- the cell values (on or off) of the identification codes c1 to c8 of each number (No.) are indicated by “1” and “0”.
- FIG. 10 shows an example of the arrangement of identification codes in the projection pattern (an example of the arrangement of areas made up of cells of 5 rows and 5 columns including each identification code).
- Each square in FIG. 10 corresponds to a region MB made up of cells of 5 rows and 5 columns.
- the numbers in each square indicate the identification code numbers (No.) in FIG.
- the same identification codes are arranged in the vertical direction, and the identification codes are No. 1 in the horizontal direction from left to right. 0 to No. No. 55 are arranged in order. After No. 55 (on the right side), no. 0 is arranged, and thereafter the same arrangement is repeated (periodic arrangement). In the center of the projection pattern, no. 28 is located.
- the arrangement of the cells in the on state and the off state is the center of the projection pattern (projection). It is located at the midpoint of the vertical direction of the pattern and is point-symmetric with respect to the center of the light spot MK in the area MB including the identification code No. 28.
- the identification codes associated with the adjacent light spots in the horizontal direction there is always only one place where the on state / off state is changed (the place where the on state is switched to the off state or the place where the off state is switched to the on state). It has become.
- a projection pattern When a projection pattern is generated using a diffraction grating, it is more asymmetrical to project a point-symmetric pattern (a pattern in which a pattern rotated 180 degrees with respect to the center matches the original pattern) with respect to the center of the projection pattern.
- a point-symmetric pattern a pattern in which a pattern rotated 180 degrees with respect to the center matches the original pattern
- the shape of the diffraction grating can be simplified, and the cost for designing and manufacturing the diffraction grating can be reduced.
- the arrangement of the identification codes is determined so that the projection pattern is a pattern having a shape that is point-symmetric with respect to the center.
- the number of cells to be turned on is 4 or less. This prevents the identification code from appearing as bright as the light spot, making it easier to identify the light spot, reducing the number of cells that are turned on, and improving the brightness of the cells that are turned on. This is to facilitate time pattern detection.
- the projection pattern position can be easily identified. From this viewpoint, it is desirable that the number of cells constituting each identification code to be turned on is smaller.
- the projection pattern has point symmetry, and in order to realize the number of 56 identification codes necessary for discrimination between patterns, a combination in which four cells are turned on must also be used as the identification code. There is.
- the projection pattern formed when the pattern light is projected onto a plane that is not perpendicular to the optical axis of the projection unit 22 is a quadrilateral other than a rectangle, and the rows and columns of the light spots are not parallel to each other. The distance is also not uniform. In the projection pattern formed when the pattern light is projected onto the curved surface, the rows and columns of the light spots are not linear. If the surface on which the pattern light is projected has irregularities, steps, etc., the relationship between the projection angles of each light spot (for example, the order of arrangement from the smallest) and the relationship between the incident angles of each light spot ( For example, the order in the case of arranging from the smallest may not match, and “replacement” may occur.
- the 8-bit identification code itself does not include an amount of information that can identify the number of columns by itself, but, for example, even if there is a change between the light spots, a deviation from the original position (order) of each light spot is not possible.
- the range of the change of the “value” of the identification code in the example shown in FIG. 10, the area MB composed of cells of 5 rows and 5 columns, 56 positions. It is possible to identify the position of the light spot, and by identifying the original position, it is possible to identify the column of the light spot to which the identification code is added.
- the above-mentioned “replacement” occurs when the imaging unit 11 and the projection unit 22 are arranged at different positions in the horizontal direction, and the imaging unit 11 and the projection unit 22 are arranged at the same position in the vertical direction. Therefore, the above-mentioned interchange does not occur in the vertical direction. Therefore, the vertical position (order) in the projection pattern can be determined by detecting the order in the captured image. Can do. Therefore, a code for identifying the order in the vertical direction is not necessary.
- FIG. 11 shows a configuration example of the distance information generation unit 18.
- the distance information generation unit 18 shown in FIG. 11 includes a binarization unit 81, a spot area extraction unit 82, an identification code reading unit 83, a storage unit 84, an effectiveness confirmation unit 85, and a projection angle estimation unit 86. And an incident angle calculation unit 87 and a distance calculation unit 88.
- the binarization unit 81 binarizes the projection image component output from the separation unit 16 and outputs a binary projection image.
- the spot area extraction unit 82 extracts a spot area MA (area of 4 rows and 4 columns in FIG. 8) centered on each light spot from the projection image.
- a spot area MA area of 4 rows and 4 columns in FIG. 8
- a group of 4 ⁇ 4 cells consisting of cells in the off state is searched.
- the group of four dots of 2 rows and 2 columns in the center is regularly arranged at regular intervals in the projection pattern, and therefore the same characteristics are obtained in an image obtained by imaging. Is a condition.
- the captured image does not always have the same interval due to the curvature, unevenness, step, etc. of the surface of the subject, so the pattern matching based on the similarity is performed to extract the spot area MA.
- the identification code reading unit 83 reads the identification code DC from the identification code area adjacent to the extracted spot area MA.
- the read identification code value is stored in the storage unit 84.
- the identification code reading unit 83 reads the identification code adjacent to each spot area MA, the identification code part adjacent to the upper side (DCa ′) and the identification code part adjacent to the left side (DCb ′) have already been read before that.
- the values are stored in the storage unit 84, the values of the identification code portions may be read from the storage unit 84.
- the processing for the spot areas adjacent to the upper side and the left side has been completed, and thus stored in the storage unit 84 as described above. It is possible to use the identified identification code.
- the validity confirmation unit 85 checks the validity of the identification code read by the identification code reading unit 83. As a result of the check, when there is a question about the validity (when there is no reliability), the read identification code is not used for the subsequent processing. As shown in FIG. 12, the validity is determined by the first identification code portion DCa adjacent to the lower side of each spot area MA, the second identification code portion DCb adjacent to the right side, and adjacent to the upper side. The identification code portion DCa ′ and the identification code portion DCb ′ adjacent on the left side are used.
- the states (4 bits) of the four cells constituting the identification code portion DCa are represented by c1 to c4, and the 4 bits constituting the identification code portion DCb are represented by symbols c5 to c8.
- the states (4 bits) of the four cells constituting the identification code portion DCa ′ are represented by c1 ′ to c4 ′
- the 4 bits constituting the identification code portion DCb ′ are represented by codes c5 ′ to c8 ′. Since c1 ′ to c4 ′ are associated with the light spots MK arranged in the upward direction, they should be the same values as c1 to c4.
- the above condition (b) is (B1) “The number of different portions (changed portions) between the second portions c5 to c8 and c5 ′ to c8 ′ of the identification code is 1 or less between adjacent light spots in the left-right direction, and is adjacent in the vertical direction.
- the first portions c1 to c4 and c1 ′ to c4 ′ of the identification code are the same as each other.
- (B2) “For each light spot, there are no more than one place (change place) different between the identification code portions c5 to c8 adjacent on the right side and the identification code portions c5 ′ to c8 ′ adjacent on the left side.
- the identification code portions c1 to c4 adjacent on the side and the identification code portions c1 ′ to c4 ′ adjacent on the upper side are the same ” In other words.
- the projection angle estimation unit 86 receives the identification code reading result from the identification code reading unit 83, receives the validity determination result D85 from the validity confirmation unit 85, and further receives the contents of the table of FIG. 9 from the control unit 30.
- the data (information indicating the relationship between each identification code and the position in the projection pattern) Sdp and the information Spa indicating the correspondence between the position on the projection pattern and the projection angle are obtained, and based on these, the projection angle of each light spot is obtained.
- Estimate ⁇ If the above information, that is, the data Sdp indicating the contents of the table of FIG. 9 and the information Spa indicating the correspondence between the position on the projection pattern and the projection angle are given from the control unit 30, It may be held in a memory (not shown).
- the projection angle estimation unit 86 When it is determined that the reading result by the identification code reading unit 83 is not valid by the validity checking unit 85, the projection angle estimation unit 86 does not estimate the projection angle based on the reading result. When the validity check unit 85 determines that the reading result obtained by the identification code reading unit 83 is valid, the projection angle estimation unit 86 estimates the projection angle based on the reading result.
- the value of the read identification code DC is the identification code No. in the table of FIG. 0-No. 55 is determined (ie, it is determined which position in the pattern is attached to the light spot), and based on the determination result, the position of the light spot in the projection pattern is determined. Identify what is. If the identification code does not exist in the table of FIG. 9 (when it does not match any of the identification codes attached to the light spots in the pattern), it is regarded as a reading error and is not used for determining the position of the light spot.
- the projection angle ⁇ is obtained based on the information Spa (given from the control unit 30) indicating the relationship between the identified position and the projection angle.
- the incident angle calculation unit 87 determines the position of the light spot in the imaging plane, and the light spot based on the axial direction and the angle of view of the imaging unit. Is calculated. Information Szv representing the axial direction and the angle of view is supplied from the control unit 30.
- the distance calculation unit 88 generates a light spot based on the projection angle ⁇ estimated by the projection angle estimation unit 86, the incident angle ⁇ calculated by the incident angle calculation unit 87, and the baseline length Lpc supplied from the control unit 30. Calculate the distance to the surface of the subject being projected.
- the distance R from the imaging unit to the subject surface (spot SP) on which the light spot is formed is based on the distance Z to the base line BL obtained by Expression (2) and the incident angle ⁇ .
- R Z / cos ⁇ (3) It can ask for.
- FIG. 13 shows a processing procedure of the distance information generation unit 18 described above.
- the binarized projection image pattern is binarized (ST101)
- the spot area MA is extracted from the binarized projection image pattern (ST102).
- identification codes DC, DCa ′, and DCb ′ are read from identification code areas adjacent to each spot area MA (ST103).
- the validity of the identification code is determined (ST104). If the identification code is valid, the projection angle ⁇ is estimated (ST105).
- the incident angle ⁇ is calculated (ST106).
- the distance is calculated using the projection angle ⁇ and the incident angle ⁇ (ST107). It is determined whether or not steps ST103 to ST107 have been performed for all spot areas MA in the captured projection image (ST108), and the process ends when all the processes have been performed. If it is not valid in step ST104 (No), the process proceeds to step ST108.
- the distance to the subject surface (spot SP) on which each light spot is projected can be obtained.
- the projection pattern of the light spot in the captured image can be obtained by using the identification code. Therefore, the position in the projection pattern, the estimation of the projection angle, and the calculation of the distance can be accurately performed with a small amount of calculation.
- the display processing unit 19 displays distance information associated with the background image.
- 14A and 14B show examples of output images of the display processing unit 19.
- FIG. 14A shows a background image
- FIG. 14B shows an image with distance information.
- an image with distance information an image in which luminance or color is assigned to the distance is displayed.
- an image in which the background image is expressed by luminance and the distance is expressed by color is displayed.
- an object existing in the imaging space is recognized, and an image in which character information representing the distance of the object is superimposed and displayed on the background image is output.
- two display screens may be used to display the background image of FIG. 14 (a) on one side and the image with distance information shown in FIG. 14 (b) on the other side.
- the image with distance information shown in FIG. 14 (b) may be displayed alternately, or the one selected by the user's operation may be displayed. In this case, it is desirable to display the image with distance information synchronously with the same angle of view and the number of pixels as the background image.
- the distance can be obtained with a small amount of calculation.
- the design of the diffraction grating can be facilitated by reducing the cost by using a point-symmetric pattern.
- each identification code DC by limiting the number of cells that are turned on within each identification code DC to a maximum of four, it is easier to identify the light spot and reduce the number of dots (configured by cells in the on state) It is possible to improve dot brightness and facilitate pattern detection during camera shooting.
- the obtained identification codes c1 to c8 are valid if the condition (b) is satisfied, and the obtained identification codes c1 to c8 are valid if the condition (b) is not satisfied. It is determined that c8 is not effective (low reliability).
- (C) There is always only one place to change between the identification codes of adjacent light spots in the horizontal direction (the place from the on state to the off state or from the off state to the on state) " By utilizing this, it is possible to compare the identification code between adjacent light spots and determine the validity of the reading result of the identification code.
- the identification code consisting of the lower and right cells c1 to c8 of each light spot MK is assigned to the identification code consisting of the lower and right cells c1 to c8 on the left side of the light spot MK ′.
- the identification code consisting of the lower and right cells c1 to c8 of each light spot is different by 1 bit from the identification code consisting of the lower and right cells c1 to c8 on the left. " If the above condition is satisfied, the obtained identification codes c1 to c8 are effective, and if the above condition (c1) is not satisfied, the obtained identification codes c1 to c8 are not effective (low reliability). It is also possible to judge.
- the first portion DCa composed of cells that are adjacent to the spot area on the lower side and arranged in the horizontal direction is adjacent to the spot area on the right side and aligned in the vertical direction.
- the one including the second part DCb made of cells is used, only one of the first part and the second part may be used.
- the first part may be adjacent to the spot area on the upper side, and the second part may be adjacent to the spot area on the left side. Further, it may include only one of a part made up of cells arranged in the horizontal direction like the first part and a part made up of cells arranged in the vertical direction like the second part.
- the imaging unit 11 and the projection unit 22 are arranged so as to be aligned in the horizontal direction, and an identification code that enables identification from other light spots at different positions in the horizontal direction is used.
- the imaging unit 11 and the projection unit 22 may be arranged so as to be aligned in the vertical direction, and an identification code that enables identification from other light spots at different positions in the vertical direction may be used.
- identification that can be distinguished from other light spots at different positions. Use code.
- first direction the direction in the projection pattern corresponding to the direction in which the imaging unit 11 and the projection unit 22 are arranged
- the direction perpendicular to the first direction is referred to as a second direction
- (d1) “The second portion of the identification code between adjacent light spots in the first direction ( c5 to c8, c5 ′ to c8 ′) are one or less different locations (change locations), and the first portion (c1 to c4, c1) of the identification code between adjacent light spots in the second direction ' ⁇ C4') are the same " (D2) “For each light spot, an identification code portion (c5 to c8) adjacent on one side in the first direction and an identification code portion (c5′ ⁇ ) adjacent on the other side in the first direction. c8 ') is different in one place or less (changed place) between the identification code portion (c1 to c4) adjacent on one side in the second direction and the other side in the second direction.
- FIG. 16 The configuration of the second embodiment of the present invention is shown in FIG. However, the distance information generator 18 shown in FIG. 16 is used.
- the distance information generation unit 18 of FIG. 16 is generally the same as that of FIG. 11, but differs in that a distance range calculation unit 89 is added.
- the distance range calculation unit 89 calculates the size of the spot area MA in the captured image and the size of the dots forming the light spot in the spot area (relatively high brightness in each cell in the image obtained by imaging).
- the range of the distance to the subject surface on which the light spot is projected is estimated on the basis of the ratio to the subject surface).
- the distance calculation unit 88 determines that the result of the distance calculation based on the projection angle is low in reliability. .
- FIGS. 17A and 17B show the ratio of the size of the dot DT constituting the spot area MA and the light spot MK depending on the distance of the subject.
- the dots that make up the pattern are formed by a single laser beam that is collimated by a collimating lens. That does not change.
- the size of the entire pattern is determined by the mutual distance between the diffracted dots. Since the diffracted lights forming different dots are projected in a radial pattern from the projection unit 22 rather than in parallel, the distance between the dots increases as the distance to the subject increases.
- the resolution number of pixels
- the number of horizontal pixels and the number of vertical pixels of the image sensor in the horizontal direction and the vertical direction respectively correspond to the number of dot positions (cells) in the projection pattern in the horizontal direction and the number in the vertical direction. It is necessary to increase the number sufficiently, for example, about 10 times or more is desirable.
- FIG. 18 shows a processing procedure in the distance information generation unit 18 according to the second embodiment.
- the process of FIG. 18 is the same as the process of FIG. 13, but steps ST111 to ST114 are added.
- step ST111 the distance range to the subject surface on which the dots are formed is estimated from the ratio between the size of the dots and the size of the spot area MA.
- step ST112 it is determined whether or not the distance obtained in step ST107 is within the distance range obtained in step ST111. If it is within the distance range (Yes), it is determined that the distance obtained in ST107 is valid (ST113), and processing based on the determination result is performed.
- step ST108 if not within the distance range (No), It is determined that the distance obtained in step ST107 is not valid (ST114), processing based on the determination result is performed, and then the process proceeds to step ST107.
- the second embodiment is the same as the first embodiment.
- the range of distance that the subject can take is narrowed down to reduce false detection of the light spot and the subject distance detection accuracy. It is possible to improve. Note that the modification described in the first embodiment can also be applied to the second embodiment.
- FIG. FIG. 19 is a block diagram showing a configuration of an image generation apparatus according to Embodiment 3 of the present invention.
- the illustrated image generation apparatus is generally the same as that shown in FIG. 1, but differs in that the imaging unit 11 includes an optical filter 13.
- the optical filter 13 is a spectral transmission filter that transmits incident light with a predetermined wavelength characteristic. Specifically, it has a characteristic that the transmittance of the visible light wavelength body is lower than the transmittance of the wavelength of the projected pattern light.
- the wavelength of the projected pattern light that is, 100% is transmitted in a wavelength band centered on 830 nm which is the emission wavelength of the laser light source 23, and only a predetermined transmittance is obtained in the visible light wavelength band. Transmits and does not transmit in other wavelength bands.
- the predetermined transmittance in the visible light wavelength band is set based on the spectral distribution and brightness of the ambient light in the imaging space JS, and the projection pattern projected by the projection unit 22, particularly the light intensity of each dot.
- the ambient light component distributed with a large power centering on the visible light wavelength band is dominant, and the wavelength centered on 830 nm. Since the ratio of the pattern light component of only the band is very small, it is difficult to extract the pattern light component.
- the optical filter 13 is provided, and the pattern light component is transmitted through 100% to capture an image, while the ambient light component is attenuated and imaged to separate or extract the projection image component from the image signal. make it easier.
- the projected image component and the background component are in a ratio of, for example, 1:64 in the imaging signal D11 output from the imaging element 14, 256 is obtained by the 8-bit imaging signal D15 obtained by A / D conversion of the imaging signal D11.
- About four gradations out of the gradations represent the projected image component. If the difference between the total of the projected image component and the background component and the background component is equal to or greater than one gradation, it can be separated in principle, but a margin is given in consideration of the influence of noise, and the above difference is a predetermined level. It is set to have a value equal to or greater than the logarithm.
- the number of gradations is the number of gradations obtained by adding the number of gradations of the expected noise component to one gradation necessary for the lowest source when there is no noise.
- the transmittance of the optical filter 13 in the visible light wavelength band is set so that the projected image component and the background component have a ratio of, for example, 1:64.
- the light intensity of each dot of the projection pattern is determined by the light emission power of the laser light source 23 and the number of dots formed.
- the ambient light has an illuminance for each wavelength based on the spectral radiation characteristic of the light source of the environmental light, and the amount of light can be calculated based on the spectral radiation characteristic.
- the case where the emission wavelength of the laser light source 23 is 830 nm has been described as an example. good.
- the imaging element is provided with the optical filter having the spectral transmission characteristic, the pattern light component can be extracted while minimizing the influence of the dominant ambient light component, and even in a high illumination environment.
- the distance can be measured and the distance information associated with the image can be obtained.
- the background image and distance information are acquired by a single image sensor, the angle of view of the background image by the ambient light and the angle of view of the image having the distance information are completely the same, and the distance of the subject in the image is accurately determined Can grasp.
- the optical filter 13 has a transmission characteristic in the visible light wavelength band and the wavelength of the projection pattern light, and the transmittance in the visible light wavelength band is set lower than the transmittance of the wavelength of the projection pattern light. Therefore, among the outputs of the image sensor 14, the ambient light component in the visible light wavelength band that prevents the extraction of the pattern light component can be reduced, and the pattern light component can be extracted with high accuracy. The distance can be measured even under, and the distance information associated with the image can be obtained.
- an image pickup element in which RGB pixels are arranged in a Bayer array is used as the image pickup element 14.
- a monochrome image pickup element operates in the same manner as described above and has the same effect. can get.
- Embodiment 4 FIG. The configuration of the fourth embodiment of the present invention is shown in FIG. 19 as in the third embodiment. However, the image generation unit 17 shown in FIG. 21 is used.
- the image generation unit 17 illustrated in FIG. 21 includes a pixel addition unit 74 in addition to the video signal processing unit 72 similar to that illustrated in FIG.
- an imaging signal (a signal representing a background component) that does not include the projection image component output from the separation unit 16 is applied to the input terminal 71.
- the pixel addition unit 74 applies the R, G, and B pixels output from the output terminal 65 of the separation unit 16 to a pixel value of each of the R, G, and B pixels (target pixel) in a Bayer array.
- a signal having a multiplied pixel value is output by adding pixel values of pixels of the same color that are located around the pixel.
- FIGS. 22A to 22C show pixels added to the target pixel.
- the smallest squares each represent a pixel.
- the target pixel is the R pixel RR34 as shown in FIG. 22A
- the pixel RR12 at the left of the second column on the second row
- the pixel RR32 at the same column on the second row
- the second row The top two pixels right RR52, the same row two columns left pixel RR14, the same row two columns right pixel RR54, the second row down two columns left pixel RR16, the second row down pixel RR36
- Eight pixels of the pixel RR56 on the right in the second column and two rows below are added. Therefore, the addition result NRR34 is expressed by the following equation.
- NRR34 RR12 + RR32 + RR52 + RR14 + RR34 + RR54 + RR16 + RR36 + RR56
- the pixel of interest is a G pixel GB33 as shown in FIG. 22B
- a peripheral pixel a pixel GB31 in the same column on the second row, a pixel GR22 on the left one column on the first row, Pixel GR42 on the right by one column on the row, pixel GB13 on the left by two columns on the same row, pixel GB53 on the right by two columns on the same row, pixel GR24 on the left by one column on the first row, pixel on the right by one column on the bottom Eight pixels of GR44 and pixel GB35 in the same column under two rows are added. Therefore, the addition result NGB33 is expressed by the following equation.
- NGB33 GB31 + GR22 + GR42 + GB13 + GB33 + GB53 + GR24 + GR44 + GB35
- the pixel of interest is a B pixel BB43 as shown in FIG. 22C
- a peripheral pixel a pixel BB21 that is two columns left on the second row, a pixel BB41 in the same column on the second row, Pixel BB61, two columns right on the second row, pixel BB23 two columns on the left in the same row, pixel BB63, two columns on the right in the same row, pixel BB25 on the second row, two columns on the left, and pixels in the same column on the second row
- the addition result NBB43 is expressed by the following equation.
- NBB43 BB21 + BB41 + BB61 + BB23 + BB43 + BB63 + BB25 + BB45 + BB65
- the above addition process is a process of mixing the peripheral pixels in the same frame with the target pixel. Since the peripheral pixels generally have the same pixel value as the target pixel, the signal component is increased. Has the effect of doubling. For example, when the pixel values of the surrounding eight pixels are added to each pixel of interest as described above (assuming that the surrounding pixels have the same pixel value as the pixel of interest), the addition result is 9 times the pixel value. However, as a result of adding (mixing) peripheral pixels, the resolution (still resolution) is lowered.
- pixels in the same position as the target pixel in different frames may be added.
- the preceding and following frames are not limited to the immediately preceding frame and the immediately following frame, but may be the immediately preceding predetermined number of frames and the immediately following predetermined number of frames. If pixels at the same position in different frames are added, the signal component can be enhanced while avoiding a decrease in still resolution, which is particularly effective when there is little movement of the image. However, the motion blur increases in the case of an image with intense motion.
- both the peripheral pixels in the same frame and the pixels at the same position in different frames may be added to the target pixel, and further, the pixels around the pixels at the same position in different frames may be added. Also good. By doing so, the multiplication factor of the signal component can be further increased.
- the video signal processing unit 72 uses a video signal obtained by adding gradation correction processing, noise reduction processing, contour correction processing, white balance adjustment processing, signal amplitude adjustment processing, color correction processing, and the like to the output signal of the pixel addition unit 74 as a background. An image is output from the output terminal 73.
- the transmittance in the visible light wavelength band is the number of pixels added by the pixel adder 74. Is set based on For example, the transmittance is set to a smaller value as the number of added pixels in the pixel adding unit 74 is larger. More specifically, the transmittance of the optical filter 13 is set to the reciprocal of the number of added pixels of the pixel adding unit 74. For example, when the pixel addition unit 74 performs 9-pixel addition, the transmittance is set to 1/9 (11.1%).
- the pixel adder 74 is only required to restore the brightness of the image by minimizing the decrease in resolution and compensating for the attenuation of the visible light wavelength band by the optical filter 13, and the signal processing different from the addition of the peripheral pixels is also applied. Is possible. For example, by detecting the correlation of pixel values and selecting and adding pixels with strong correlation, it is possible to reduce the decrease in resolution as a background image.
- the pixel adding unit 74 adds the surrounding eight pixels to the target pixel.
- the visible light wavelength band transmittance may be set to increase or decrease the number of peripheral pixels to be added.
- a monochrome image sensor may be used as the image sensor 14 instead of the R, G, and B pixels arranged in a Bayer array. Even in this case, the same operation is performed and the same effect can be obtained.
- a monochrome image sensor is used in the fourth embodiment, pixels at closer positions can be added, and an image with less reduction in resolution can be generated as a background image.
- the image generation unit 17 includes the pixel addition unit 74 that mixes the surrounding pixels and multiplies the signal component, the transmittance of the optical filter 13 in the visible light wavelength band is changed to the pattern.
- the transmittance of the optical filter 13 in the visible light wavelength band is changed to the pattern.
- the signal component is multiplied by mixing the pixels at the same position as the target pixel in the frames before and after the frame including the target pixel, the reduction in still resolution can be suppressed. Even a subject having a complicated contour can restore a bright and clear background image while minimizing a decrease in resolution.
- an optical filter 13 having a transmittance in the visible light wavelength band set to the reciprocal of the number of added pixels (multiplier) of the pixel adder 74 is used, a background having substantially the same brightness as that before attenuation by the optical filter 13 is used. Images can be restored. At the same time, it is possible to obtain an imaging signal with a reduced ambient light component in the visible light wavelength band that hinders the extraction of the pattern light component, and it is possible to extract the pattern light component with high accuracy and bright illuminance. Distance can be measured even in an environment, and distance information associated with an image can be obtained.
- a laser is used as the light source of the projection unit 22.
- another light source such as an LED is used instead, the incident light characteristics of the diffraction grating are satisfied. The operation is similar and the same effect can be obtained.
- the projection unit 22 is configured to project the pattern formed on the diffraction grating 26 by the laser light source 23.
- the laser beam is scanned two-dimensionally at high speed ( A configuration in which a pattern is projected by scanning the entire field of view within one frame period operates in the same manner and provides the same effect.
- a diffraction grating is used as an element for forming pattern light.
- the transmission type described in paragraphs 0003 and 0004 of Patent Document 3 is used.
- a spectral device having a projection pattern such as a computer generated hologram (Computer Generated Hologram) is used, the same operation and the same effect can be obtained.
- an image generation apparatus that can acquire distance information to an object existing in an imaging space in association with a captured image. Moreover, only one image sensor may be used.
- the image generation apparatus of the present invention can be applied to intrusion monitoring in monitoring applications because it can simultaneously acquire an intruder image and distance, for example.
- the image generation apparatus of the present invention can also be applied to driving assistance based on obstacle detection in front of or behind the vehicle, for example, parking assistance.
Abstract
Description
撮像空間に所定波長のパターン光を投写する投写部と、
前記撮像空間を撮像する撮像部と、
前記撮像部による撮像で得られた撮像信号のうち、パターン光投写時に得られた撮像信号とパターン光非投写時に得られた撮像信号の差分を求めることでパターン投写像成分を分離する分離部と、
前記分離部で分離された前記投写像成分に基づいて距離情報を生成する距離情報生成部とを備え、
前記距離情報生成部は、前記投写像成分で表される投写像内の光スポットの配列と、予め記憶されている、投写パターン内における各光スポットの位置と投写角との関係から、撮像された投写パターン内の各光スポットの投写角を特定し、特定された投写角に基づいて、光スポットが投写されている被写体の表面までの距離を求め、
前記投写部から投写されるパターン光は、各光スポットに付随して識別コードを構成する、各々オン状態又はオフ状態の複数のセルを含み、
前記距離情報生成部は、前記識別コードに基づいて、該識別コードが付随した光スポットの、投写パターン内における位置を特定し、
各光スポットに付随する識別コードは、投写パターン内の第1の方向に並んだ複数のセルから成り、当該光スポットに対し、投写パターン内の第2の方向における一方の側に配置された第1の部分と、前記第2の方向に並んだ複数のセルから成り、当該光スポットに対し前記第1の方向における一方の側に配置された第2の部分とを有し、
前記識別コードは、
前記第1の方向において隣接する光スポット間で、前記識別コードの前記第2の部分を構成するセルのオン状態からオフ状態へ、又はオフ状態からオン状態へ切り替わる変更箇所が一箇所以下であり、
前記第2の方向において隣接する光スポット間で、前記識別コードの前記第1の部分が互いに同じである。
撮像空間に所定波長のパターン光を投写する投写部と、
前記撮像空間を撮像する撮像部と、
前記撮像部による撮像で得られた撮像信号のうち、パターン光投写時に得られた撮像信号とパターン光非投写時に得られた撮像信号の差分を求めることでパターン投写像成分を分離する分離部と、
前記分離部で分離された前記投写像成分に基づいて距離情報を生成する距離情報生成部とを備え、
前記距離情報生成部は、前記投写像成分で表される投写像内の光スポットの配列と、予め記憶されている、投写パターン内における各光スポットの位置と投写角との関係から、撮像された投写パターン内の各光スポットの投写角を特定し、特定された投写角に基づいて、光スポットが投写されている被写体の表面までの距離を求め、
前記投写部から投写されるパターン光は、各光スポットに付随して識別コードを構成する、各々オン状態又はオフ状態の複数のセルを含み、
前記距離情報生成部は、前記識別コードに基づいて、該識別コードが付随した光スポットの、投写パターン内における位置を特定し、
前記識別コードは、投写パターン内の第1の方向において隣接する光スポット間で該識別コードを構成するセルのオン状態からオフ状態へ、又はオフ状態からオン状態へ切り替わる変更箇所が一箇所のみとなるように定められている。
また、撮像画像中の各光スポットについて投写されたパターン内における位置を特定することができ、少ない演算量で被写体までの距離情報を取得することができる。
図1は本発明の実施の形態1における画像生成装置の構成を示すブロック図である。図示の画像生成装置は、画像取得部10と、パターン光生成部20と、制御部30とを有する。
画像取得部10は、撮像部11を有し、パターン光生成部20は、投写部22を有する。
本発明の画像生成装置は、撮像によって得た情報に基づいて、撮像された被写体OJ1、OJ2の各部までの距離を求め、画像情報と、画像の各部についての距離情報と得るものである。
上記の投写パターンにおける横方向は、基線BLの方向、即ち水平方向に対応し、縦方向は、水平方向に直交する方向に対応する。
ここで、投写角φは、図3に示すように、基線BLと光スポットSPを含む平面内において、基線BLに垂直な線と、投写部22と光スポットSPを結ぶ線の成す角である。
一方入射角θは、図3に示すように、基線BLと光スポットSPを含む平面内において、基線BLに垂直な線と、撮像部11と光スポットSPを結ぶ線の成す角である。
投写部22からの投写角φは、投写部22の構成によって予め定まっており、従って既知である。
この場合、
(a) 「投写部22におけるそれぞれの光スポットの投写角の大小関係(例えば小さいものから並べたときの順序)と、撮像部11におけるそれぞれの光スポットの入射角の大小関係(小さいものから並べたときの順序)が同じである」
と言う条件が満たされ、かつそのことが分かっている場合には、そのことに基づいて、撮像部11で撮像した光スポットの各々の投写角が分かる。
駆動部21は、制御部30により制御されて、レーザー光源23を発光させ、レーザー光源23から出射されたレーザー光はコリメートレンズ24で平行光にされ、アパーチャ25で所定のビーム径にされる。
回折格子26は所定の投写パターンを生成するためのパターン光を撮像空間JSに投写する。
撮像素子14は入射像を光電変換した撮像信号を出力する。撮像素子14は例えばR、G、Bの画素がベイヤ型に配列されたものであり、撮像信号としては、R、G、Bの信号が出力される。
制御部30は例えば、撮像部11の撮像素子14に対し、撮像モードやフレームレート、露光時間などを制御し、表示処理部19に対し、表示モードや、距離情報の表示モードなどの設定を行う。制御部30はまた、A/D変換部15に対して動作のタイミングを制御するための信号を供給する。制御部30はさらに、パターン光生成部20及び画像取得部10に対し、動作モードの設定を行う。
より具体的には、制御部30は、撮像部11に対し、予め定められたフレーム周波数で撮像を繰り返すよう制御を行うとともに、レーザー光源23が1フレームおきに発光状態、非発光状態になるように駆動部21を制御し、さらに、レーザー光源23が発光状態にあるか、非発光状態にあるかを示す信号Snfを分離部16に供給する。
各フレームの撮像のタイミングは、制御部30により制御される。
図5において入力端子60にA/D変換部15の出力(デジタル撮像信号)D15が供給される。
フレーム遅延部61は、入力端子60に供給されたデジタル撮像信号D15を、1フレーム期間遅延させてフレーム遅延撮像信号D61を出力する。
第2のフレームPS2では、差分演算部62は、そのときの撮像信号D15(第2のフレームでの撮像信号D15、即ち、パターン光が投写されている状態での撮像部出力D11をデジタル変換することで得られた信号D15)から、遅延部61の出力D61(第1のフレームPS1の撮像信号)を減算し、減算結果(差分)D62を出力する(図6(c))。
第4のフレームPS4では、第2のフレームPS2と同様、差分演算部62は、そのときの撮像信号D15から、フレーム遅延部61の出力D61を減算し、減算結果(差分)D62を出力する。
以下、同様の処理が繰り返され、1フレーム期間おきに、背景成分のみの画像と投写像成分のみの画像が出力される。
また、上記の分離部16のフレーム遅延部61、差分演算部62、及びスイッチ64による分離の処理は、R、G、Bの各々について別箇に行われ、分離の結果得られた投写像のR、G、B成分に対して補間(ベイヤ配列のため各画素について欠落している色成分の補間)を行ってすべての画素のすべての色成分(R、G、B成分)を生成し、各画素についてのR、G、B成分を合成することで当該画素についての輝度成分を生成し、該輝度成分が投写像成分として出力される。
最小のマスはドット位置乃至セルと呼ばれるものであり、投写パターンにおいて、オン(光が照射された状態)又はオフ(光が照射されない状態)を制御可能な最小単位である。例えば、投写範囲内に縦方向480行、横方向650列のセルが形成される。光が照射された状態のセルによりドットが構成される。
2行2列の領域の周囲の上下各1行、左右各1列はオフ状態のセル(光が照射されないセル)からなる領域であり、この領域と2行2列の領域を含む4行4列の領域をスポット領域MAと言う。
第1の部分DCa及び第2の部分DCbの各セルはオン状態(照射された状態)又はオフ状態(照射されない状態)のいずれかを取ることが可能であり、このオン、オフの組み合わせにより、8ビットの識別コードDCが構成される。各光スポットMKに付随した識別コードDCは、当該光スポットMKの識別のために用いられる。
投写パターンの全体は、上記の4行4列のスポット領域MAに、識別コードDC及びセルcbrを加えた、5行5列のセルからなる領域MBの繰り返しで構成される。
各光スポットMKに付随した識別コードDCは、当該光スポットMKが投写パターンに含まれる多数の光スポットのうちのどれであるかの判定に用いられるものである。
図10に示す例では、垂直方向には同じ識別コードが並べられ、水平方向には、左から右へ、識別コードがNo.0からNo.55まで順に並べられ、No.55の次(右側)には、再びNo.0が配置され、以下同様の配置の繰り返し(周期的な配置)となっている。
また、投写パターンの中心に、No.28が位置するように配置されている。
また、水平方向において隣接する光スポットに付随する識別コード間では、オン状態/オフ状態の変更箇所(オン状態からオフ状態へ切り替わる箇所又はオフ状態からオン状態へ切り替わる箇所)が必ず一箇所のみとなっている。
この点を考慮し、本実施の形態では投写パターンとして、中心に対し点対称となる形状のパターンとなるように識別コードの配列を決定している。
図11に示される距離情報生成部18は、2値化部81と、スポット領域抽出部82と、識別コード読取部83と、記憶部84と、有効性確認部85と、投写角推定部86と、入射角算出部87と、距離算出部88とを有する。
スポット領域MAの抽出のためには、一定の間隔で中心部に2行2列の4つのドット(オン状態のセルによって構成される)と、その周囲(上下各1行、左右各1列)のオフ状態のセルから成る4行4列のセルの群を探索する。また、中心部の2行2列の4つドットの群は、投写パターンにおいては、規則的に等間隔に配置されたものであるので、撮像で得られた画像においても同様の特性があることが条件となる。但し、被写体の表面の曲がり、凹凸、段差などにより、撮像画像においては、完全に等間隔になるとは限らないため、類似度に基づくパターンマッチング等を行って、スポット領域MAの抽出を行う。
有効性の判定には、図12に示すように、各スポット領域MAに対して下側に隣接する第1の識別コード部分DCa、右側に隣接する第2の識別コード部分DCb、上側に隣接する識別コード部分DCa’、左側に隣接する識別コード部分DCb’を用いる。
さらに識別コード部分DCa’を構成する4つのセルの状態(4ビット)をc1’~c4’で表し、識別コード部分DCb’を構成する4ビットを符号c5’~c8’で表す。
c1’~c4’は上方向に並んだ光スポットMKに付随したものであるので、c1~c4と同じ値のはずである。
一方、c5’~c8’は左方向に並んだ光スポットMKに付随したものであるので、上記した、「隣接する識別コード間では、オン状態/オフ状態の変更箇所が必ず一箇所のみ」であるとの条件から、c5~c8と同じ、又は1ビットだけ異なるものであるはずである。
そこで、
(b) 「c1~c4がc1’~c4’と同じであり、かつc5~c8がc5’~c8’と同じ、又は1ビットだけ異なる」
と言う条件が満たされれば、得られた識別コードc1~c8は有効であり、上記の条件(b)が満たされなければ、得られた識別コードc1~c8は有効ではない(信頼性が低い)と判断する。
上記の条件(b)は、
(b1) 「左右方向で隣接する光スポット間で、識別コードの第2の部分c5~c8、c5’~c8’間の異なる箇所(変更箇所)が1箇所以下であり、上下方向で隣接する光スポット間で、識別コードの第1の部分c1~c4、c1’~c4’が互いに同じである」
と言い換えることもでき、
(b2) 「各光スポットについてその右側において隣接する識別コード部分c5~c8と左側において隣接する識別コード部分c5’~c8’との間で異なる箇所(変更箇所)が1箇所以下であり、下側において隣接する識別コード部分c1~c4と上側において隣接する識別コード部分c1’~c4’とが同じである」
と言い換えることもできる。
ビットc1’~c8’について有効であるとの判断がなされていない状態では、c1~c8とc1’~c8’のいずれかが有効でないとの判断を保留し、他の識別コードとの比較結果をも利用して総合的に判断することとしても良い。
識別コード読取部83での読み取り結果について、有効性確認部85で有効であるとの判断がなされた場合には、投写角推定部86は、該読み取り結果に基づいて投写角の推定を行う。
識別コードが、図9の表に存在しないものであれば(パターン内の光スポットに付された識別コードのいずれとも合致しないときは)、読み取りエラーとし、光スポットの位置判定には用いない。
光スポットの、投写パターン上の位置が特定できたら、特定された位置と投写角との関係を示す情報Spa(制御部30から与えられる)に基づいて投写角φを求める。
Z=Lpc/(tanφ-tanθ) (1)
の関係から求めることができる。式(1)は、図3において、
Z・tanφ-Z・tanθ=Lpc (2)
の関係があることから得られる。
R=Z/cosθ (3)
により求めることができる。
まず2値化された投写像パターンを2値化し(ST101)、
次に、2値化した投写像パターンから、スポット領域MAの抽出を行う(ST102)。
次に、各スポット領域MAに隣接した識別コード領域から識別コードDC、DCa’、及びDCb’を読み取る(ST103)。
次に、識別コードの有効性の判定を行う(ST104)。
識別コードが有効であれば、投写角φの推定を行う(ST105)。
次に、入射角θの算出を行う(ST106)。
次に投写角φ及び入射角θを用いて、距離の算出を行う(ST107)。
ステップST103~ST107を、撮像された投写像中のすべてのスポット領域MAについて行われたか否かの判定を行い(ST108)、すべてについて処理が行われたら、終了する。
なお、ステップST104で有効でなければ(No)、ステップST108に進む。
図14(a)及び(b)は、表示処理部19の出力画像の例を示す。
図14(a)は、背景画像を表し、
図14(b)は、距離情報付き画像を表す。
距離情報付き画像としては、距離に輝度又は色を割り当てたものが表示される。例えば、背景画像を輝度で表現し、距離を色で表現した画像が表示される。あるいは、撮像空間に存在する物体を認識して、当該物体の距離を表す文字情報を背景画像に重畳表示した画像を出力する。
また、例えば、2つの表示画面を用いて、一方に図14(a)の背景画像を表示し、他方に図14(b)に示す距離情報付き画像を表示しても良く、一つの表示画面で、図14(a)に示す背景画像と、図14(b)に示す距離情報付き画像を交互に表示しても良く、ユーザによる操作により選択された方を表示するようにしても良い。この場合、距離情報付き画像は、背景画像と同じ画角、画素数で、同期して表示するのが望ましい。
また、回折格子を用いてパターンを投写する際に、点対称のパターンとすることにより、回折格子の設計が容易化でき、コスト低減を図ることが可能である。
(c) 水平方向において隣接する光スポットの識別コード間の変更箇所(オン状態からオフ状態へ、又はオフ状態からオン状態に切り替わる箇所)が必ず一箇所のみである」
ことを利用して、識別コードを隣接する光スポット間で比較し、識別コードの読取結果の有効性を判定することが可能である。
(c1) 「各光スポットの下側及び右側のセルc1~c8から成る識別コードが、その左隣の光スポットの下側及び右側のセルc1~c8から成る識別コードに対して1ビットだけ異なる」
と言う条件が満たされれば、得られた識別コードc1~c8は有効であり、上記の条件(c1)が満たされなければ、得られた識別コードc1~c8は有効ではない(信頼性が低い)と判断することとしても良い。
要するに、撮像部11と投写部22を並べた方向(撮像部11及び投写部22が設置された空間内の第1の方向)において、異なる位置にある他の光スポットとの識別が可能な識別コードを用いれば良い。
(d1) 「第1の方向において隣接する光スポット間で、識別コードの第2の部分(c5~c8、c5’~c8’)間の異なる箇所(変更箇所)が1箇所以下であり、第2の方向において隣接する光スポット間で、識別コードの第1の部分(c1~c4、c1’~c4’)が互いに同じである」、
(d2) 「各光スポットについて、その第1の方向の一方の側において隣接する識別コード部分(c5~c8)と、前記第1の方向の他方の側において隣接する識別コード部分(c5’~c8’)との間で異なる箇所(変更箇所)が1箇所以下であり、第2の方向の一方の側において隣接する識別コード部分(c1~c4)と前記第2の方向の他方の側において隣接する識別コード部分(c1’~c4’)とが同じである」
となり、上記した図15を参照して上記した条件(c)を一般化して言えば、(e) 「第1の方向において互いに隣接する光スポット間で、それらの識別コードの変更箇所が一箇所のみである」
となり、いずれの場合にもこれらの条件が満たされるか否かに応じて、識別コードの読み取り結果が有効か否かの判断をすることになる。
本発明の実施の形態2の構成は実施の形態1と同じく図1によって示される。但し、距離情報生成部18として、図16に示すものが用いられる。
図16の距離情報生成部18は、図11のものと概して同じであるが、距離範囲算出部89が付加されている点で異なる。
距離範囲算出部89は、撮像画像中のスポット領域MAの大きさと、スポット領域内の、光スポットを形成するドットの大きさ(撮像により得られた画像中の、各セル内の比較的高輝度の部分の大きさ)との割合に基づいて、光スポットが投射されている被写体表面までの距離の範囲を推定する。
距離算出部88は、投写角に基づいて算出した距離が距離範囲算出部89で算出した距離範囲内のものでないときは、投写角に基づく距離の計算の結果を信頼性の低いものと判断する。
投写パターンをレーザー光源と回折格子の組み合わせを用いて投写する場合、パターンを構成するドットは、コリメートレンズによって平行化されたレーザー光一点で形成されるため、被写体の距離によらずドット自体の大きさは変わらない。一方パターン全体の大きさは、回折されたドット間の相互距離によって決まる。異なるドットを形成する回折光同士は平行ではなく投写部22から放射状に投写されているため、被写体までの距離が遠くなるに従いドット間の距離が広がる。そのため被写体の距離に応じて、投写パターンの大きさ、従って、スポット領域MAの大きさとドットDTの大きさの比率が変化し、比率を計測することによって被写体のとりうる距離の範囲を求める。
スポット領域MAの大きさに対する各ドットの大きさの比率の変化を検出するために、撮像素子の解像度(画素の数)を十分大きくする必要がある。具体的には、撮像素子の水平方向及び垂直方向の画素の水平方向の数及び垂直方向の数が、それぞれ投写パターン内のドット位置(セル)の水平方向の数及び垂直方向の数に対して十分に多くする必要があり、例えば10倍程度以上あることが望ましい。
図18の処理は、図13の処理と同様であるが、ステップST111~ST114が付加されている。
ステップST111では、ドットの大きさとスポット領域MAの大きさの比から、ドットが形成された被写体表面までの距離範囲を推定する。
ステップST112では、ステップST107で求めた距離がステップST111で求めた距離範囲内か否かの判定を行う。距離の範囲内であれば(Yes)、ST107で求めた距離が有効と判断して(ST113)、判断結果に基づく処理を行い、その後ステップST108に進み、距離範囲内でなければ(No)、ステップST107で求めた距離が有効でないと判断して(ST114)、判断結果に基づく処理を行い、その後、ステップST107に進む。
上記以外の点では、実施の形態2は実施の形態1と同じである。
なお、実施の形態1について記載した変形例は、実施の形態2にも適用可能である。
図19は本発明の実施の形態3における画像生成装置の構成を示すブロック図である。
図示の画像生成装置は、図1に示すものと概して同じであるが、撮像部11が光学フィルタ13を備えている点で異なる。
光学フィルタ13の分光透過特性の一例を図20に示す。図20に示す特性の場合、投写されるパターン光の波長、即ち、レーザー光源23の発光波長である830nmを中心とした波長帯で100%透過し、可視光波長帯では、所定の透過率だけ透過し、それ以外の波長帯では透過しない。
可視光波長帯の所定の透過率は、撮像空間JSの環境光の分光分布と明るさと、投写部22により投写される投写パターン、特に各ドットの光の強さに基づいて設定される。
投写像成分と背景成分の合計と、背景成分との差が1階調分以上であれば、原理的には分離できるが、ノイズの影響を考慮して余裕を与え、上記の差が所定階調数以上の値を有するものとなるように設定される。即ち、上記の階調数は、ノイズがない場合に最低源必要な1階調に、予想されるノイズ成分の階調数を加えた階調数とされる。
光学フィルタ13の可視光波長帯の透過率は、投写像成分と背景成分が例えば1:64の割合となるように設定される。
本発明の実施の形態4の構成は実施の形態3と同じく図19によって示される。但し、画像生成部17として、図21に示されるものが用いられる。
図21において、入力端子71に分離部16から出力される投写像成分を含まない撮像信号(背景成分を表す信号)が印加される。
画素加算部74は、分離部16の出力端子65から出力されるR画素、G画素、B画素がベイヤ配列されたR、G、Bの画素の各々(注目画素)の画素値に対し、当該画素の周辺に位置し、同じ色の画素の画素値を加算することで、増倍された画素値を有する信号を出力する。
注目画素が図22(a)に示されるようにR画素RR34である場合には、周辺の画素として、2行上で2列左の画素RR12、2行上で同じ列の画素RR32、2行上で2列右の画素RR52、同じ行で2列左の画素RR14、同じ行で2列右の画素RR54、2行下で2列左の画素RR16、2行下で同じ列の画素RR36、及び2行下で2列右の画素RR56の、8つの画素が加算される。
従って、加算結果NRR34は下の式で表される。
NRR34=RR12+RR32+RR52
+RR14+RR34+RR54
+RR16+RR36+RR56
以上注目画素がRR34である場合について説明したが、他の位置のR画素についても、同様の配置の周辺画素を加算する。
従って、加算結果NGB33は下の式で表される。
NGB33=GB31+GR22+GR42
+GB13+GB33+GB53
+GR24+GR44+GB35
以上注目画素がGB33である場合について説明したが、他の位置のG画素についても、同様の配置の周辺の画素を加算する。
従って、加算結果NBB43は下の式で表される。
NBB43=BB21+BB41+BB61
+BB23+BB43+BB63
+BB25+BB45+BB65
以上注目画素がBB43である場合について説明したが、他の位置のR画素についても、同様の配置の周辺画素を加算する。
例えば各注目画素に対して上記のように周辺の8個の画素の画素値を加算する場合(仮に周囲の画素が注目画素と同じ画素値を有するとすれば)、加算結果は、注目画素の画素値の9倍となる。
但し、周辺の画素を加算(混合)する結果、解像度(静止解像度)は低下する。
ここで、前後のフレームとは直前の1フレーム及び直後の1フレームに限らず、直前の所定数のフレーム、及び直後の所定数のフレームであっても良い。
異なるフレームの同一位置の画素を加算することとすれば、静止解像度の低下を避けながら、信号成分を増強することができ、画像の動きが少ない場合に特に有効である。
但し、動きの激しい映像の場合に動きぼけが多くなる。
そのようにすれば信号成分の増倍率を一層大きくすることができる。
例えば、画素値の相関性を検出して、相関の強い画素を選択して加算することにより、背景画像として、解像度の低下をより少なくすることもできる。
Claims (19)
- 撮像空間に所定波長のパターン光を投写する投写部と、
前記撮像空間を撮像する撮像部と、
前記撮像部による撮像で得られた撮像信号のうち、パターン光投写時に得られた撮像信号とパターン光非投写時に得られた撮像信号の差分を求めることでパターン投写像成分を分離する分離部と、
前記分離部で分離された前記投写像成分に基づいて距離情報を生成する距離情報生成部とを備え、
前記距離情報生成部は、前記投写像成分で表される投写像内の光スポットの配列と、予め記憶されている、投写パターン内における各光スポットの位置と投写角との関係から、撮像された投写パターン内の各光スポットの投写角を特定し、特定された投写角に基づいて、光スポットが投写されている被写体の表面までの距離を求め、
前記投写部から投写されるパターン光は、各光スポットに付随して識別コードを構成する、各々オン状態又はオフ状態の複数のセルを含み、
前記距離情報生成部は、前記識別コードに基づいて、該識別コードが付随した光スポットの、投写パターン内における位置を特定し、
各光スポットに付随する識別コードは、投写パターン内の第1の方向に並んだ複数のセルから成り、当該光スポットに対し、投写パターン内の第2の方向における一方の側に配置された第1の部分と、前記第2の方向に並んだ複数のセルから成り、当該光スポットに対し前記第1の方向における一方の側に配置された第2の部分とを有し、
前記識別コードは、
前記第1の方向において隣接する光スポット間で、前記識別コードの前記第2の部分を構成するセルのオン状態からオフ状態へ、又はオフ状態からオン状態へ切り替わる変更箇所が一箇所以下であり、
前記第2の方向において隣接する光スポット間で、前記識別コードの前記第1の部分が互いに同じである
画像生成装置。 - 前記距離情報生成部は、
前記分離部で得られた前記投写像成分に含まれる、各光スポットに付随した識別コードを読み取る識別コード読取部と、
前記識別コード読取部で読み取られた、各光スポットに付随した識別コードの前記第2の部分が、当該光スポットに対して前記第1の方向において隣接する光スポットに付随した識別コードの前記第2の部分に対して前記変更箇所が一箇所以下であり、かつ前記識別コード読取部で読み取られた、各光スポットに付随した識別コードの前記第1の部分が、当該光スポットに対して前記第2の方向において隣接する光スポットに付随した識別コードの前記第1の部分と同じであると言う条件が満たされるか否かを判定し、これに基づいて読み取られた識別コードが有効であるか否かを判定する有効性確認部と、
前記有効性確認部により有効であると判定された識別コードに基づいて各光スポットの投写角の推定を行う投写角推定部と
を有する
請求項1に記載の画像生成装置。 - 撮像空間に所定波長のパターン光を投写する投写部と、
前記撮像空間を撮像する撮像部と、
前記撮像部による撮像で得られた撮像信号のうち、パターン光投写時に得られた撮像信号とパターン光非投写時に得られた撮像信号の差分を求めることでパターン投写像成分を分離する分離部と、
前記分離部で分離された前記投写像成分に基づいて距離情報を生成する距離情報生成部とを備え、
前記距離情報生成部は、前記投写像成分で表される投写像内の光スポットの配列と、予め記憶されている、投写パターン内における各光スポットの位置と投写角との関係から、撮像された投写パターン内の各光スポットの投写角を特定し、特定された投写角に基づいて、光スポットが投写されている被写体の表面までの距離を求め、
前記投写部から投写されるパターン光は、各光スポットに付随して識別コードを構成する、各々オン状態又はオフ状態の複数のセルを含み、
前記距離情報生成部は、前記識別コードに基づいて、該識別コードが付随した光スポットの、投写パターン内における位置を特定し、
前記識別コードは、投写パターン内の第1の方向において隣接する光スポット間で該識別コードを構成するセルのオン状態からオフ状態へ、又はオフ状態からオン状態へ切り替わる変更箇所が一箇所のみとなるように定められている
画像生成装置。 - 各光スポットに付随する識別コードは、投写パターン内の第1の方向に並んだ複数のセルから成り、当該光スポットに対し、投写パターン内の第2の方向における一方の側に配置された第1の部分と、前記第2の方向に並んだ複数のセルから成り、当該光スポットに対し前記第1の方向における一方の側に配置された第2の部分とを有する
請求項3に記載の画像生成装置。 - 前記距離情報生成部は、
前記分離部で得られた前記投写像成分に含まれる、各光スポットに付随した識別コードを読み取る識別コード読取部と、
前記識別コード読取部で読み取られた、各光スポットに付随した識別コードが、前記第1の方向において隣接する光スポットに付随した識別コードに対して前記変更箇所が一箇所のみであるという条件が満たされるか否かを判定し、これに基づいて読み取られた識別コードが有効であるか否かを判定する有効性確認部と、
前記有効性確認部により有効であると判定された識別コードに基づいて各光スポットの投写角の推定を行う投写角推定部と
を有する
請求項3又は4に記載の画像生成装置。 - 各光スポットに付随する前記識別コードの各々内でオン状態となるセルの数が4個以下である
請求項1から5のいずれか1項に記載の画像生成装置。 - 前記投写パターンに含まれるすべての光スポットに付随する識別コードを構成する、オン状態及びオフ状態のセルの配列が前記投写パターンの中心に対して点対称となるように前記識別コードが定められている
請求項1から6のいずれか1項に記載の画像生成装置。 - 前記光スポットは、2行2列のオン状態のセルで構成され、各光スポットと、その周囲のセルとにより、スポット領域が形成されている
請求項1から7のいずれか1項に記載の画像生成装置。 - 前記距離情報生成部は、撮像された投写像内における、前記スポット領域の大きさと、前記スポット領域内の各セルに位置するドットの大きさの比から、当該光スポットが投写された被写体までの距離の範囲を算出し、
前記距離情報生成部は、前記投写角に基づいて算出した前記被写体までの距離が、前記スポット領域の大きさと前記ドットの大きさとの比に基づいて定められた距離の範囲に入らないときは、前記投写角に基づいて算出した前記距離を前記投写角に基づく前記距離の算出結果を、無効なものとして処理する
請求項8に記載の画像生成装置。 - 前記撮像部と前記投写部とが空間内の第1の方向に並んで配置され、
前記投写パターン内の前記第1の方向は、前記空間内の第1の方向に対応する方向である
請求項1から9のいずれか1項に記載の距離情報付き画像生成装置。 - 前記撮像部は、可視光波長帯とパターン光の波長とに透過特性を有し、可視光波長帯の透過率が、前記パターン光の波長の透過率より低い光学フィルタを備える
請求項1から10のいずれか1項に記載の画像生成装置。 - 前記分離部は、前記撮像部による撮像で得られた撮像信号のうち、パターン光投写時に得られた撮像信号とパターン光非投写時に得られた撮像信号の差分を求めることで前記投写像成分を分離する
請求項11に記載の画像生成装置。 - 前記光学フィルタの透過特性は、前記差分が前記撮像信号の所定階調数以上の値を有するものとなるように定められる
請求項12に記載の画像生成装置。 - 前記分離部で分離された前記背景成分から背景画像を生成する背景画像生成部をさらに有する
請求項11に記載の画像生成装置。 - 前記背景画像生成部は、周辺の画素の画素値を加算して信号成分を増倍する画素加算部を備える
請求項14に記載の画像生成装置。 - 前記画素加算部は、各画素に対して同じフレーム内の周辺に位置する画素の画素値を加算する
請求項15に記載の画像生成装置。 - 前記画素加算部は、各画素に対して、当該画素が含まれるフレームの前後に位置するフレーム内の、当該画素と同じ位置にある画素の画素値を加算する
請求項15に記載の画像生成装置。 - 前記光学フィルタの可視光波長帯の透過率は、前記画素加算部の加算画素数が多いほど小さな値に定められる
請求項15から17のいずれか1項に記載の画像生成装置。 - 前記光学フィルタの可視光波長帯の透過率は、前記画素加算部の加算画素数の逆数に等しい
請求項18に記載の画像生成装置。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE112013005574.0T DE112013005574B4 (de) | 2012-11-21 | 2013-11-20 | Bilderzeugungsvorrichtung |
JP2014548595A JP5808502B2 (ja) | 2012-11-21 | 2013-11-20 | 画像生成装置 |
US14/424,391 US9709387B2 (en) | 2012-11-21 | 2013-11-20 | Image generation device for acquiring distances of objects present in image space |
CN201380060555.9A CN104813141B (zh) | 2012-11-21 | 2013-11-20 | 图像生成装置 |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012255427 | 2012-11-21 | ||
JP2012-255482 | 2012-11-21 | ||
JP2012-255427 | 2012-11-21 | ||
JP2012255482 | 2012-11-21 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014080937A1 true WO2014080937A1 (ja) | 2014-05-30 |
Family
ID=50776117
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2013/081272 WO2014080937A1 (ja) | 2012-11-21 | 2013-11-20 | 画像生成装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US9709387B2 (ja) |
JP (1) | JP5808502B2 (ja) |
CN (1) | CN104813141B (ja) |
DE (1) | DE112013005574B4 (ja) |
WO (1) | WO2014080937A1 (ja) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016124261A1 (de) * | 2015-02-06 | 2016-08-11 | Siemens Aktiengesellschaft | Vorrichtung und verfahren zur sequentiellen, diffraktiven musterprojektion |
JP2017086803A (ja) * | 2015-11-17 | 2017-05-25 | 富士フイルム株式会社 | 計測装置、内視鏡システム、及び計測方法 |
JPWO2016199323A1 (ja) * | 2015-06-09 | 2017-11-02 | 三菱電機株式会社 | 画像生成装置、画像生成方法、及びパターン光生成装置 |
WO2018070210A1 (ja) * | 2016-10-14 | 2018-04-19 | オムロン株式会社 | 3次元測定装置および3次元測定方法 |
JP2018538107A (ja) * | 2015-09-23 | 2018-12-27 | イースト カロライナ ユニバーシティ | 発光体を使用して物体距離及びターゲット寸法を測定する方法、システム、及びコンピュータプログラム製品 |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9779546B2 (en) | 2012-05-04 | 2017-10-03 | Intermec Ip Corp. | Volume dimensioning systems and methods |
US10007858B2 (en) | 2012-05-15 | 2018-06-26 | Honeywell International Inc. | Terminals and methods for dimensioning objects |
US10321127B2 (en) | 2012-08-20 | 2019-06-11 | Intermec Ip Corp. | Volume dimensioning system calibration systems and methods |
US20140104413A1 (en) | 2012-10-16 | 2014-04-17 | Hand Held Products, Inc. | Integrated dimensioning and weighing system |
US10228452B2 (en) | 2013-06-07 | 2019-03-12 | Hand Held Products, Inc. | Method of error correction for 3D imaging device |
US9823059B2 (en) | 2014-08-06 | 2017-11-21 | Hand Held Products, Inc. | Dimensioning system with guided alignment |
US10810715B2 (en) | 2014-10-10 | 2020-10-20 | Hand Held Products, Inc | System and method for picking validation |
US10775165B2 (en) | 2014-10-10 | 2020-09-15 | Hand Held Products, Inc. | Methods for improving the accuracy of dimensioning-system measurements |
US9897434B2 (en) | 2014-10-21 | 2018-02-20 | Hand Held Products, Inc. | Handheld dimensioning system with measurement-conformance feedback |
US9786101B2 (en) | 2015-05-19 | 2017-10-10 | Hand Held Products, Inc. | Evaluating image values |
US20160377414A1 (en) | 2015-06-23 | 2016-12-29 | Hand Held Products, Inc. | Optical pattern projector |
US9835486B2 (en) | 2015-07-07 | 2017-12-05 | Hand Held Products, Inc. | Mobile dimensioner apparatus for use in commerce |
US20170017301A1 (en) | 2015-07-16 | 2017-01-19 | Hand Held Products, Inc. | Adjusting dimensioning results using augmented reality |
US10249030B2 (en) | 2015-10-30 | 2019-04-02 | Hand Held Products, Inc. | Image transformation for indicia reading |
US10025314B2 (en) | 2016-01-27 | 2018-07-17 | Hand Held Products, Inc. | Vehicle positioning and object avoidance |
US10339352B2 (en) | 2016-06-03 | 2019-07-02 | Hand Held Products, Inc. | Wearable metrological apparatus |
CN106441268B (zh) * | 2016-08-30 | 2019-09-10 | 西安小光子网络科技有限公司 | 一种基于光标签的定位方法 |
US10909708B2 (en) * | 2016-12-09 | 2021-02-02 | Hand Held Products, Inc. | Calibrating a dimensioner using ratios of measurable parameters of optic ally-perceptible geometric elements |
US11047672B2 (en) | 2017-03-28 | 2021-06-29 | Hand Held Products, Inc. | System for optically dimensioning |
US10733748B2 (en) | 2017-07-24 | 2020-08-04 | Hand Held Products, Inc. | Dual-pattern optical 3D dimensioning |
EP3662406B1 (en) * | 2017-08-01 | 2023-11-22 | Apple Inc. | Determining sparse versus dense pattern illumination |
US10591276B2 (en) | 2017-08-29 | 2020-03-17 | Faro Technologies, Inc. | Articulated arm coordinate measuring machine having a color laser line probe |
US10699442B2 (en) | 2017-08-29 | 2020-06-30 | Faro Technologies, Inc. | Articulated arm coordinate measuring machine having a color laser line probe |
CN108038839B (zh) * | 2017-11-23 | 2022-01-11 | 宁波大学 | 一种流水生产线上双绞线绞距实时检测方法 |
CA2996173C (en) * | 2018-02-23 | 2020-02-18 | Librestream Technologies Inc. | Image processing system for inspecting object distance and dimensions using a hand-held camera with a collimated laser |
CN108426567A (zh) * | 2018-03-22 | 2018-08-21 | 西安闻泰电子科技有限公司 | 被拍摄对象距离显示方法、装置及系统 |
US10584962B2 (en) | 2018-05-01 | 2020-03-10 | Hand Held Products, Inc | System and method for validating physical-item security |
JP7115057B2 (ja) * | 2018-06-20 | 2022-08-09 | オムロン株式会社 | 計測システムおよび計測方法 |
CN109194953B (zh) * | 2018-08-15 | 2021-03-02 | 瑞声光学解决方案私人有限公司 | 空间颜色和分辨率测量装置及测量方法 |
US11639846B2 (en) | 2019-09-27 | 2023-05-02 | Honeywell International Inc. | Dual-pattern optical 3D dimensioning |
CN111461040A (zh) * | 2020-04-07 | 2020-07-28 | 武汉华星光电技术有限公司 | 电子设备及其光学指纹识别模组 |
DE102021206336A1 (de) * | 2021-06-21 | 2022-12-22 | Robert Bosch Gesellschaft mit beschränkter Haftung | Vorrichtung und Verfahren zur optischen Analyse einer Oberfläche |
CN114764821B (zh) * | 2022-04-19 | 2023-07-28 | 极米科技股份有限公司 | 移动物体检测方法、装置、电子设备和存储介质 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004191092A (ja) * | 2002-12-09 | 2004-07-08 | Ricoh Co Ltd | 3次元情報取得システム |
JP2007256090A (ja) * | 2006-03-23 | 2007-10-04 | Nissan Motor Co Ltd | 車両用環境認識装置及び車両用環境認識方法 |
JP2008067362A (ja) * | 2006-08-10 | 2008-03-21 | Nissan Motor Co Ltd | 画像処理装置および画像処理方法 |
JP2008188196A (ja) * | 2007-02-05 | 2008-08-21 | Sanyo Electric Co Ltd | 撮像装置 |
JP2010101683A (ja) * | 2008-10-22 | 2010-05-06 | Nissan Motor Co Ltd | 距離計測装置および距離計測方法 |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0726828B2 (ja) | 1986-04-18 | 1995-03-29 | 株式会社トプコン | 形状測定装置 |
JPH0424590A (ja) | 1990-05-18 | 1992-01-28 | Tokimec Inc | 移動体識別装置 |
JPH0687377A (ja) | 1992-09-04 | 1994-03-29 | Yazaki Corp | 車両周辺監視装置 |
JP2000111490A (ja) | 1998-10-05 | 2000-04-21 | Toyota Motor Corp | 塗装面の検出装置 |
JP4315545B2 (ja) | 1999-10-19 | 2009-08-19 | Hoya株式会社 | 3次元画像検出システムと3次元画像検出装置 |
JP3714063B2 (ja) | 1999-10-19 | 2005-11-09 | 富士ゼロックス株式会社 | 3次元形状計測装置 |
JP4410603B2 (ja) | 2004-05-21 | 2010-02-03 | 日本放送協会 | 3次元情報検出方法及びその装置 |
JP4442539B2 (ja) | 2005-09-27 | 2010-03-31 | パナソニック電工株式会社 | 空間情報の検出装置 |
WO2006046519A1 (ja) | 2004-10-25 | 2006-05-04 | Matsushita Electric Works, Ltd. | 空間情報検出装置 |
JP4821192B2 (ja) | 2005-07-06 | 2011-11-24 | 大日本印刷株式会社 | 計算機ホログラム光学素子 |
CN100562065C (zh) * | 2005-08-15 | 2009-11-18 | 索尼株式会社 | 摄像装置、降噪装置、降噪方法 |
US20070091174A1 (en) | 2005-09-30 | 2007-04-26 | Topcon Corporation | Projection device for three-dimensional measurement, and three-dimensional measurement system |
CN101496032B (zh) | 2006-02-27 | 2011-08-17 | 普莱姆传感有限公司 | 使用斑纹解相关的距离映射的方法及设备 |
JP2011169701A (ja) | 2010-02-17 | 2011-09-01 | Sanyo Electric Co Ltd | 物体検出装置および情報取得装置 |
EP2395769B1 (en) * | 2010-06-11 | 2015-03-04 | Nintendo Co., Ltd. | Image display program, image display system, and image display method |
JP5163713B2 (ja) | 2010-08-24 | 2013-03-13 | カシオ計算機株式会社 | 距離画像センサ及び距離画像生成装置並びに距離画像データ取得方法及び距離画像生成方法 |
US9207070B2 (en) * | 2012-05-24 | 2015-12-08 | Qualcomm Incorporated | Transmission of affine-invariant spatial mask for active depth sensing |
-
2013
- 2013-11-20 JP JP2014548595A patent/JP5808502B2/ja active Active
- 2013-11-20 WO PCT/JP2013/081272 patent/WO2014080937A1/ja active Application Filing
- 2013-11-20 DE DE112013005574.0T patent/DE112013005574B4/de active Active
- 2013-11-20 US US14/424,391 patent/US9709387B2/en active Active
- 2013-11-20 CN CN201380060555.9A patent/CN104813141B/zh active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004191092A (ja) * | 2002-12-09 | 2004-07-08 | Ricoh Co Ltd | 3次元情報取得システム |
JP2007256090A (ja) * | 2006-03-23 | 2007-10-04 | Nissan Motor Co Ltd | 車両用環境認識装置及び車両用環境認識方法 |
JP2008067362A (ja) * | 2006-08-10 | 2008-03-21 | Nissan Motor Co Ltd | 画像処理装置および画像処理方法 |
JP2008188196A (ja) * | 2007-02-05 | 2008-08-21 | Sanyo Electric Co Ltd | 撮像装置 |
JP2010101683A (ja) * | 2008-10-22 | 2010-05-06 | Nissan Motor Co Ltd | 距離計測装置および距離計測方法 |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016124261A1 (de) * | 2015-02-06 | 2016-08-11 | Siemens Aktiengesellschaft | Vorrichtung und verfahren zur sequentiellen, diffraktiven musterprojektion |
JPWO2016199323A1 (ja) * | 2015-06-09 | 2017-11-02 | 三菱電機株式会社 | 画像生成装置、画像生成方法、及びパターン光生成装置 |
CN108271407A (zh) * | 2015-06-09 | 2018-07-10 | 三菱电机株式会社 | 图像生成装置、图像生成方法以及图案光生成装置 |
US10533846B2 (en) | 2015-06-09 | 2020-01-14 | Mitsubishi Electric Corporation | Image generation device, image generating method, and pattern light generation device |
CN108271407B (zh) * | 2015-06-09 | 2020-07-31 | 三菱电机株式会社 | 图像生成装置、图像生成方法以及图案光生成装置 |
DE112015006609B4 (de) | 2015-06-09 | 2021-10-21 | Mitsubishi Electric Corporation | Bilderzeugungsgerät, Bilderzeugungsverfahren und Musterlichterzeugungsgerät |
JP2018538107A (ja) * | 2015-09-23 | 2018-12-27 | イースト カロライナ ユニバーシティ | 発光体を使用して物体距離及びターゲット寸法を測定する方法、システム、及びコンピュータプログラム製品 |
JP2017086803A (ja) * | 2015-11-17 | 2017-05-25 | 富士フイルム株式会社 | 計測装置、内視鏡システム、及び計測方法 |
WO2018070210A1 (ja) * | 2016-10-14 | 2018-04-19 | オムロン株式会社 | 3次元測定装置および3次元測定方法 |
JP2018063220A (ja) * | 2016-10-14 | 2018-04-19 | オムロン株式会社 | 3次元測定装置および3次元測定方法 |
US10818030B2 (en) | 2016-10-14 | 2020-10-27 | Omron Corporation | Three-dimensional measurement apparatus and three-dimensional measurement method |
Also Published As
Publication number | Publication date |
---|---|
DE112013005574B4 (de) | 2018-10-31 |
CN104813141A (zh) | 2015-07-29 |
CN104813141B (zh) | 2017-02-22 |
JP5808502B2 (ja) | 2015-11-10 |
US20150204657A1 (en) | 2015-07-23 |
US9709387B2 (en) | 2017-07-18 |
DE112013005574T5 (de) | 2015-08-27 |
JPWO2014080937A1 (ja) | 2017-01-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5808502B2 (ja) | 画像生成装置 | |
US9900485B2 (en) | Image generation device | |
JP6355844B2 (ja) | 画像生成装置、画像生成方法、及びパターン光生成装置 | |
US10041787B2 (en) | Object detection device | |
US9383549B2 (en) | Imaging system | |
EP2175232A1 (en) | Three-dimensional shape measuring device, three-dimensional shape measuring method, three-dimensional shape measuring program, and recording medium | |
JP2002191058A (ja) | 3次元画像取得装置および3次元画像取得方法 | |
JP2008252401A (ja) | 撮像システム、撮像方法、および撮像プログラム | |
US20160360093A1 (en) | Imaging apparatus, imaging method, and storage medium | |
JP2009290268A (ja) | 撮像装置 | |
JP6190119B2 (ja) | 画像処理装置、撮像装置、制御方法、及びプログラム | |
KR102122275B1 (ko) | 배광 특성 측정 장치 및 배광 특성 측정 방법 | |
JP2002027501A (ja) | 3次元画像撮像装置および3次元画像撮像方法 | |
JP2020193957A (ja) | 測距異常を補正する距離画像生成装置 | |
JP6016000B1 (ja) | 超解像処理方法、超解像処理装置及び超解像処理システム | |
JPWO2019087895A1 (ja) | 電子内視鏡用プロセッサ及び電子内視鏡システム | |
JP2021072591A (ja) | 情報取得装置、情報取得方法及びプログラム | |
JP2017158182A (ja) | 画像読取装置 | |
JP2007279085A (ja) | 共焦点顕微鏡 | |
JP2022087645A5 (ja) | ||
JPWO2019088259A1 (ja) | 電子内視鏡用プロセッサ及び電子内視鏡システム | |
WO2015178079A1 (ja) | 撮像装置、撮像装置の制御方法、および、撮像装置の制御プログラム | |
JP2021026259A (ja) | 画像読み取り装置、画像読み取り方法、真贋判定装置 | |
JP2021052286A (ja) | 撮像装置及びその制御方法、プログラム、並びに記憶媒体 | |
JPH118798A (ja) | Ccdカメラシステムおよび画像処理方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13856638 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2014548595 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14424391 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 112013005574 Country of ref document: DE Ref document number: 1120130055740 Country of ref document: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 13856638 Country of ref document: EP Kind code of ref document: A1 |