WO2013114732A1

WO2013114732A1 - Electronic device

Info

Publication number: WO2013114732A1
Application number: PCT/JP2012/081876
Authority: WO
Inventors: 松本　慎也
Original assignee: 株式会社ザクティ
Priority date: 2012-01-30
Filing date: 2012-12-10
Publication date: 2013-08-08
Also published as: JP2013156111A

Abstract

An image pick-up device (1) uploads original positioning data (x_M, y_M, z_M), and a reference image (such as a distance image) based on the output of an image pick-up element (IS). A server device (SV) extracts from a database a plurality of evaluation target images (such as distance images) for a subject having a location near the positions (x_M, y_M, z_M), retrieves an image similar to the reference image from the extracted images, and updates the image pick-up device (1) with the position data of the subject that was associated with the similar image. A positioning data correction unit (33) corrects the original positioning data on the basis of received position data, the distance image, and image pick-up direction data.

Description

Electronics

The present invention relates to an electronic device such as an imaging device.

In an electronic apparatus such as an imaging apparatus, positional information using a global positioning system (GPS) can be recorded together with a captured image, but it is increasing. If the GPS function is used, the approximate current location of the user and the electronic device can be grasped. There is also a method in which the current location information of the mobile phone is transmitted to the server device, and the server device extracts an image related to the current location information and sends it back to the mobile phone (for example, see Patent Document 1 below).

JP 2010-277326 A

The position detection accuracy using the GPS function is limited, and errors that are unacceptable to the user may be included in the detection position. Note that the method of Patent Document 1 does not contribute to the reduction of the error.

Therefore, an object of the present invention is to provide an electronic device that can accurately correct positioning data.

A first electronic device according to the present invention generates an image pickup unit having an image pickup device that outputs an image signal of a subject, and distance data representing a distance between the electronic device and the subject using an output of the image pickup device. A distance data generation unit, a shooting direction detection unit that generates shooting direction data by detecting a shooting direction of the imaging unit, a positioning processing unit that performs positioning based on a signal from a satellite and generates original positioning data; A transmission unit that transmits a reference image based on the output of the imaging device to an external device, a reception unit that receives position data based on the reference image from the external device, received position data, the distance data, and the shooting direction data. And a positioning data correcting unit for correcting the original positioning data based on the positioning data.

This is expected to correct the positioning data accurately.

Specifically, for example, in the first electronic device, data related to the position associated with the target image corresponding to the reference image is received as the position data by the receiving unit, and the position related to the target image is related to the first electronic device. The data may include location data of an object included in the target image.

Further, for example, in the first electronic device, the transmission unit may transmit the reference image and the original positioning data to an external device, and the reception position data is a position based on the reference image and the original positioning data. It may be data.

提供 By providing the original positioning data to the external device, it is possible to reduce the processing load for determining the position data to be returned to the electronic device.

A third electronic device according to the present invention generates an image pickup unit having an image pickup device that outputs an image signal of a subject, and distance data representing a distance between the electronic device and the subject using an output of the image pickup device. A distance data generation unit, a shooting direction detection unit that generates shooting direction data by detecting a shooting direction of the imaging unit, a positioning processing unit that performs positioning based on a signal from a satellite and generates original positioning data; A transmitting unit that transmits the original positioning data to an external device, a receiving unit that receives a target image and position data based on the original positioning data from the external device, a reference image based on the target image and an output of the imaging element, and And based on the extraction position data, the distance data, and the shooting direction data, the arithmetic processing unit for extracting the position data from the received content of the receiving unit according to the comparison result And positioning data correcting unit for correcting the Kihara positioning data, characterized by comprising a.

This also expects accurate correction of positioning data.

Specifically, for example, in the second electronic device, the position data may include location data of an object included in the target image.

For example, in the second electronic device, the reception unit may receive a target image group and a position data group based on the original positioning data from the external device, and the arithmetic processing unit An image corresponding to the reference image may be extracted as the target image from the inside, and the position data associated with the target image may be extracted from the position data group.

For example, in the first or second electronic device, a distance image based on the distance data may be used as the reference image.

Further, for example, in the first or second electronic device, the imaging unit includes a distance measuring light source that irradiates light to the subject, and the distance data is generated using the distance measuring light source. May be.

Further, for example, in the first or second electronic apparatus, the imaging unit includes a plurality of imaging elements as the imaging element, a plurality of ranging light sources corresponding to the plurality of imaging elements, The distance measuring light source irradiates the subject with light having a plurality of wavelengths, and the distance data generation unit generates the distance data using the output of each imaging device depending on the reflected light from the subject. May be.

∙ As a result, more accurate distance data can be obtained with the influence of occlusion suppressed.

Also, for example, in the first or second electronic device, a warning notification may be made according to the correction amount to the original positioning data by the positioning data correction unit.

According to the present invention, it is possible to provide an electronic device capable of accurately correcting positioning data.

1 is a schematic overall block diagram of a positioning system according to an embodiment of the present invention. It is a figure which defines the X-, Y-, and Z-axis which concerns on embodiment of this invention. (A) And (b) is a figure for demonstrating the azimuth angle and attitude | position angle of an imaging device. (A)-(c) is a figure for demonstrating the light source for ranging which can be provided in an imaging part. It is a figure which shows the structure of an image file. It is an operation | movement flowchart of the positioning system which concerns on 1st Example of this invention. (A) And (b) is a figure which shows the example of the normal picked-up image and distance image which are acquired with an imaging device. It is an internal block diagram of the server apparatus based on 1st Example of this invention. It is a flowchart which shows the procedure of a similar image search process. It is a figure which shows several evaluation object image and several position data. It is a figure for demonstrating a positioning data correction process, Comprising: It is a figure which shows the positional relationship example between an imaging device and a target object. It is a figure for demonstrating a positioning data correction process, Comprising: It is a figure which shows the positional relationship example between an imaging device and a target object. (A) And (b) is a correction image figure of positioning data by positioning data correction processing. It is a figure which shows the modification of the one part step of FIG. It is an operation | movement flowchart of the positioning system which concerns on 2nd Example of this invention. It is an internal block diagram of the main control part which concerns on 2nd Example of this invention. It is an operation | movement flowchart of the positioning system which concerns on 3rd Example of this invention. (A) And (b) is a figure which shows the photography recommendation color image and photography recommendation distance image which concern on 3rd Example of this invention and are downloaded. It is a figure which shows the structure of the imaging part which concerns on 4th Example of this invention. It is a general | schematic whole block diagram of the positioning system which concerns on 5th Example of this invention. It is a figure which shows the imaging device and access point which concern on 6th Example of this invention.

Hereinafter, an example of an embodiment of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. In this specification, for simplification of description, a symbol or reference that refers to information, signal, physical quantity, state quantity, member, or the like is written to indicate information, signal, physical quantity, state quantity or Names of members and the like may be omitted or abbreviated.

FIG. 1 is a schematic block diagram showing the overall configuration of a positioning system according to an embodiment of the present invention. The positioning system includes an electronic device 1 and a server device SV that is an external device of the electronic device 1 and includes a computer or the like. The electronic device 1 and the server device SV are connected to a network NET formed including the Internet and the like, and can communicate arbitrary data via the network NET. The electronic device 1 includes each part referred to by reference numerals 11 to 21. The electronic device 1 has a photographing function. Therefore, hereinafter, the electronic device 1 is referred to as an imaging device (digital camera). An imaging device is a kind of electronic equipment. In the imaging apparatus 1, functions other than the imaging function (for example, a telephone function, an Internet connection function, an e-mail transmission / reception function, and a music playback function) may be further realized. In this case, the imaging apparatus 1 is a device other than the imaging apparatus. (For example, a mobile phone or an information terminal).

The imaging unit 11 has an optical system, an aperture, and an imaging element (solid-state imaging element) IS including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the subject using the imaging element IS. Take a photo of The image sensor IS photoelectrically converts an optical image of a subject incident through the optical system, and outputs an electrical signal obtained by the photoelectric conversion, that is, an image signal of the subject.

The positioning processing unit 12 receives a signal transmitted from a satellite that forms a global positioning system (GPS), and performs positioning of the imaging device 1 based on the received signal. The positioning of the imaging device 1 refers to processing for obtaining the position (location) of the imaging device 1, and the obtained position of the imaging device 1 includes the longitude, latitude, and altitude of the imaging device 1. However, the altitude of the imaging device 1 may not be included in the required position of the imaging device 1. The result of positioning by the positioning processing unit 12, that is, the obtained data including the longitude, latitude, and altitude of the imaging device 1 is output to the main control unit 21 as original positioning data. As shown in FIG. 2, an X axis, a Y axis, and a Z axis that are orthogonal to and intersect each other at the origin O are defined. The X and Y axes are parallel to the horizontal plane, and the Z axis is parallel to the direction of gravity. Below, the longitude, latitude, and altitude of the position (location) of an arbitrary object are expressed as coordinate values x, y, and z on the X, Y, and Z axes, respectively, and correspond to the coordinate values x, y, and z. Data indicating the position (location) of the object is referred to as position data or expressed as (x, y, z). The coordinate values x, y, and z of the imaging device 1 indicated by the original positioning data are represented by symbols x _M , y _M, and z _M, respectively. Therefore, the position of the imaging device 1 represented by the original positioning data is (x _M , y _M , z _M ). With reference to the origin O, the positive directions of the X and Y axes correspond to east and north, respectively, and the positive direction of the Z axis corresponds to the upward direction (the direction opposite to the direction of gravity).

The photographing direction detection unit 13 generates photographing direction data by detecting the photographing direction of the imaging unit 11 (that is, the optical axis direction of the imaging unit 11). The shooting direction data includes an azimuth angle θ and a posture angle φ. Considering that the imaging device 1 is disposed at the origin O, the significance of the azimuth angle θ and the attitude angle φ will be described with reference to FIGS. The azimuth angle θ is an angle representing the azimuth in the photographing direction, and is detected using an electronic compass or the like. For example, as shown in FIG. 3A, an angle formed by a line segment from the origin O toward the north and a line segment 301 from the origin O along the optical axis toward the subject can be defined as an azimuth angle θ. it can. However, it is assumed that θ = 90 ° when the line segment 301 extends west from the origin O, and θ = 270 ° when the line segment 301 extends east from the origin O. The posture angle φ is an angle that represents a posture difference of the imaging apparatus 1 as viewed from the reference posture, and is detected using a magnetic sensor, a gyro sensor, or the like. The posture of the imaging device 1 when the photographing direction is parallel to the horizontal plane is the reference posture. Therefore, as shown in FIG. 3B, the angle formed by the photographing direction (that is, the optical axis direction) and the horizontal plane can be defined as the posture angle φ. However, it is assumed that φ> 0 ° when the shooting direction has an upward component, and φ <0 ° when the shooting direction has a downward component.

The photographing direction recording unit 14 includes a semiconductor memory or the like, and temporarily records the photographing direction data generated by the photographing direction detection unit 13 as necessary. The memory unit 15 includes a semiconductor memory, a magnetic disk, or the like, and records arbitrary information generated by the imaging device 1 or acquired by the imaging device 1. Part or all of the memory unit 15 functions as the database 16. The display unit 17 is a display device having a display screen such as a liquid crystal display panel, and displays an arbitrary video under the control of the main control unit 21. The operation unit 18 includes a shutter button 18a that receives a still image shooting instruction, a zoom button 18b that receives a zoom magnification change instruction, and the like, and receives various operations from the user. The details of the operation on the operation unit 18 are transmitted to the main control unit 21. The

buttons

18 a and 18 b may be buttons on a touch panel that can be provided on the display unit 17.

The transmission unit 19 and the reception unit 20 form a communication unit. The transmission unit 19 is connected to the network NET and can transmit arbitrary information to any device other than the imaging device 1 including the server device SV. The receiving unit 20 is connected to the network NET and can receive any information from any device other than the imaging device 1 including the server device SV.

The main control unit 21 is formed by a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and comprehensively controls the operation of each part in the imaging apparatus 1. The main control unit 21 includes an image processing unit 31, a distance data generation unit 32, and a positioning data correction unit 33. The image processing unit 31, the distance data generation unit 32, or the positioning data correction unit 33 may be considered to be provided outside the main control unit 21.

The image processing unit 31 generates a color image by performing predetermined image processing (noise reduction processing, demosaicing processing, color correction processing, edge enhancement processing, etc.) on the image signal from the image sensor IS. The color image is a two-dimensional image representing the intensity and color of light from the subject, and is a kind of photographed image of the imaging unit 11. However, in order to distinguish from the distance image described later, the color image generated by the image processing unit 31 is hereinafter referred to as a normal captured image. In the present specification, for simplification of description, an image signal or image data representing an arbitrary image is also simply referred to as an image.

The distance data generation unit (distance image generation unit) 32 detects the subject distance of each subject located in the imaging region of the imaging unit 11 using the output image signal of the imaging element IS, and distance data representing the detection result Is generated. The subject distance of a certain subject refers to a distance in real space between the subject and the imaging device 1 or any imaging device that photographs the subject. The distance data generation unit 32 can generate a distance image as distance data. The distance image is a grayscale image in which the detection value of the subject distance is given to the pixel value of each pixel. A process for generating distance data and a distance image based on an output image signal of the imaging element IS is called a distance data generation process or a distance image generation process.

Any known method can be used as a method for detecting a subject distance using the output of the image sensor IS and a method for generating distance data. As shown in FIG. 4A, a distance measuring light source LT for irradiating light on a subject may be provided in the imaging unit 11, and distance data may be generated using the distance measuring light source LT. The distance measuring light source LT may be formed of a plurality of light emitting elements.

And, distance data may be generated using so-called TOF (Time Of Flight) ranging. More specifically, for example, a ranging light source LT is formed by an LED (Light Emitting Diode) that emits near-infrared light, and the emitted light of the LED is modulated at about 10 MHz (megahertz) and projected onto a subject. . The reflected light from the subject with respect to the light emitted from the LED is received by the image sensor IS. The distance data generation unit 32 can obtain the phase difference between the emitted light and the reflected light for each pixel based on the output image signal of the image sensor IS, and can calculate the subject distance from the phase difference for each pixel. Further, the emitted light of the LED may be modulated with a plurality of sine waves having different wavelengths (for example, several sine waves) and projected onto the subject. If only the phase difference for a single wavelength is viewed, there may be a plurality of distances that give the same phase (resulting in a large error in distance measurement). Ranging accuracy can be improved by projecting modulated waves having a plurality of wavelengths. If the phase difference is continuously measured by the image sensor IS, distance data can be generated in real time.

Alternatively, as shown in FIG. 4B, a laser is employed as the distance measuring light source LT, and laser light scanning is performed so that light from the laser is sequentially irradiated to each subject in the imaging region. good. In this case, reflected light from the subject with respect to the emitted light of the laser is received by the image sensor IS, and the distance data generation unit 32 can obtain distance data based on the output image signal of the image sensor IS. Further alternatively, the distance image generation process may be realized by a so-called pattern irradiation method. That is, as shown in FIG. 4C, the distance measuring light source LT is formed so that the subject is irradiated with laser light having a predetermined pattern (for example, a matrix pattern), and the imaging element IS representing the reflected light of the laser light is formed. The distance data may be obtained based on the output image signal (based on the degree of distortion of the reflected light).

The positioning data correction unit 33 corrects the original positioning data as necessary based on the data obtained by using the transmission unit 19 and the reception unit 20, and outputs the corrected original positioning data as corrected positioning data. . The coordinate values x, y, and z of the imaging device 1 indicated by the corrected positioning data are represented by symbols x _C , y _C, and z _C, respectively. That is, the position of the imaging device 1 indicated by the corrected positioning data is (x _C , y _C , z _C ). The correction method will be described in detail later.

[Image recording processing]
The main control unit 21 can perform image recording processing. In the image recording process, a normal captured image based on an image signal read from the image sensor IS in response to a pressing operation of the shutter button 18a (corresponding to a full pressing operation described later) is added to the memory unit 15 (for example, the memory unit 15). In the non-volatile memory). At this time, for example, as shown in FIG. 5, an image file including a header area and a main body area is formed in the memory unit 15, and additional data and a normal photographed image are stored in the header area and the main body area, respectively. For example, the image file can be made to conform to the file format of Exif (Exchangeable image file format). The additional data can include the positioning data of the photographing point of the normal photographed image, the focal length, the F value, the ISO sensitivity, etc. at the time of photographing the normal photographed image, and may further include the distance image. . Further, an image file in which a distance image is stored in the main body area instead of a normal captured image may be formed. The positioning data included in the additional data is original positioning data (x _M , y _M , z _M ) or corrected positioning data (x _C , y _C , z _C ). The original positioning data (x _M , y _M , z _M ) or corrected positioning data (x _C , y _C , z _C ) obtained one after another may be accumulated and recorded in the database 16 in time series. .

[Through display]
The main control unit 21 can perform through display using the display unit 17. Through display refers to a process of reading out image signals from the image sensor IS at a predetermined frame rate, sequentially generating normal captured images, and updating and displaying the sequentially generated normal captured images on the display unit 17. Note that the resolution of the normal captured image generated for the purpose of through display may be lower than the resolution of the normal captured image acquired and generated in the image recording process.

[Log path recording process]
The main control unit 21 can perform log path recording processing. The log path storing process refers to a process of acquiring positioning data periodically or intermittently and recording the sequentially acquired positioning data in the memory unit 15 (database 16) in time series. As a result, the movement route of the user who moves while holding the imaging device 1 is recorded. The positioning data acquired and recorded in the log path saving process is in principle the original positioning data (x _M , y _M , z _M ), but when the correction by the positioning data correction unit 33 is performed, the corrected positioning is performed. Data (x _C , y _C , z _C ).

[Positioning data correction processing]
The main control unit 21 can execute positioning data correction processing for correcting the positioning result of the positioning processing unit 12 using the positioning data correction unit 33. Specifically, for example, the CPU of the main control unit 21 reads the correction application software stored in the nonvolatile program memory in the memory unit 15, and the CPU executes the correction application software, thereby realizing the positioning data correction process. . However, the positioning data correction process may be realized only by hardware.

A more detailed embodiment of the positioning system based on the above-described configuration and operation will be described below. As long as there is no contradiction, two or more of the embodiments described below can be combined with each other, and the matters described in one embodiment can be applied to other embodiments.

<< First Example >>
A first embodiment of the positioning system will be described. FIG. 6 is an operation flowchart of the positioning system according to the first embodiment. First, in step S11, the image pickup apparatus 1 and the correction application software are activated by turning on the image pickup apparatus 1. Thereafter, through display is continuously executed, while the processes in steps S12 to S18 are sequentially executed. (However, the through display can be interrupted as appropriate). Here, it is assumed that the log path recording process is continuously executed after the imaging apparatus 1 is activated (the same applies to other examples described later).

In step S12, the positioning processing unit 12 and the shooting direction detection unit 13 acquire original positioning data (x _M , y _M , z _M ) and shooting direction data (θ, φ). At this time, the main control unit 21 may acquire a focal length to be included in the additional data (see FIG. 5).

In step S13, the image processing unit 31 acquires a normal captured image from the output image signal of the imaging element IS, and the distance data generation unit 32 acquires a distance image by a distance image generation process. The obtained normal photographed image is used for through display or recorded by image recording processing. The shutter button 18a can be pressed in two stages. The operation of pushing the shutter button 18a by half is called a half-pressing operation, and the operation of pushing the shutter button 18a completely is called a full-pressing operation. In step S13, the image processing unit 31 and the distance data generating unit 32 generate a normal captured image and a distance image based on an image signal read from the image sensor IS in response to a half-press operation or a full-press operation. Also good. When the shutter button 18a is a button on the touch panel, the above-described half-press operation or full-press operation is an operation of touching the shutter button 18a on the touch panel with an operating body (finger or operation pen), or a shutter button on the touch panel. After touching 18a with the operation body, the operation body can be replaced with an operation of releasing the operation body from the touch panel. Assume that the normal captured image and the distance image acquired in step S13 are the images 310 and 320 in FIGS. 7A and 7B, respectively. The normal captured image 310 and the distance image 320 are images obtained by capturing an object SUB that is one of the subjects. 7A and 7B, the point SUB _P represents the center of gravity of the object SUB in the normal captured image 310 and the distance image 320.

In step S <b> 14, the main control unit 21 sets the distance image acquired in step S <b> 13 as a reference image, and under the control of the main control unit 21, the transmission unit 19 together with the reference image capture position ( That is, the original positioning data (x _M , y _M , z _M ) at the time of capturing the reference image is uploaded (transmitted). Upload refers to transmitting arbitrary information from the transmission unit 19 to the server device SV (or an arbitrary site or the like) via the network NET.

In step S15, the server device SV executes a similar image search process based on the reference image and the original positioning data uploaded in step S14, thereby determining target object position data (x _T , y _T , z _T ). To do. Ideally, the object position data (x _T , y _T , z _T ) accurately indicates the location of the object SUB included in the reference image (the same applies to other examples described later).

The similar image search process will be described with reference to FIGS. FIG. 8 is an internal block diagram of the server device SV _A as the server device SV according to the first embodiment. FIG. 9 is a flowchart showing the procedure of the similar image search process. The similar image search process includes the processes of steps S31 to S34. The server device SV _A includes each part referred to by reference numerals 51 to 54. However, all or part of the database 51, connected to the network NET, it may be provided to devices other than the server device SV _A. FIG. 10 shows an image group 340 composed of a plurality of evaluation target images and a position data group 341 composed of a plurality of position data. The image group 340 and the position data group 341 are actually stored in the database 51 after being stored in a plurality of image files conforming to a file format such as Exif, for example. The i-th evaluation object image in the image group 340 is referred to by a symbol Q _i (i is an integer). Position data is associated with the evaluation target image for each evaluation target image. The position data associated with the evaluation target image Q _i is represented by (x _i , y _i , z _i ). The position data (x _i , y _i , z _i ) represents the location of the object included in the evaluation target image Q _i .

For the sake of concreteness and simplification of the description, it is assumed that the reference image includes only one object, and each evaluation object image includes only one object. As described above, when the reference image is a distance image captured by the imaging apparatus 1 (see step S14 in FIG. 6), each evaluation target image is also a distance image for the corresponding target object.

The original positioning data (x _M , y _M , z _M ) transmitted from the imaging device 1 is given to the image limitation selection unit 52. In step S <b> 31, the image limitation selection unit 52 narrows down the evaluation target images to be supplied to the similarity evaluation unit 53 in the image group 340 based on the original positioning data (x _M , y _M , z _M ). That is, the selection unit 52 supplies the similarity evaluation unit 53 with only the image of the object having a location near the position (x _M , y _M , z _M ) in the image group 340. Part of 340 is selected and selected, and the plurality of selected evaluation target images are supplied to the similarity evaluation unit 53. Note that the shooting direction data may also be given to the selection unit 52 by uploading, and the selection may be performed using the shooting direction data (further narrowing may be performed using the shooting direction data). Further, only the reference image may be uploaded in step S14 of FIG. 6, and the original positioning data may not be uploaded. However, in this case, narrowing down and selection in step S31 (FIG. 9) are not performed.

Assume that the evaluation target images Q ₁ to Q _n are selected and supplied to the similarity evaluation unit 53 (n is an integer of 2 or more). In step S32, the similarity evaluation unit 53 performs similarity evaluation processing for each evaluation object image for the evaluation object images Q ₁ to Q _n . In the similarity evaluation process, the similarity between the reference image and the evaluation target image is evaluated based on the reference image and the evaluation target image. The evaluation target image Q _i is more similar to the reference image as the evaluation target image Q _i has a higher similarity. The degree of similarity determined for the evaluation target image _{Q i} represented by the symbol SIM _i. Specifically, for example, in the similarity evaluation process, the similarity evaluation unit 53 includes a reference image and the evaluation image feature quantity of each of the reference image and the evaluation object image Q _i from the image data of the target image Q _i (object The shape, depth, edge, histogram, etc.) of each object are extracted, and the similarity SIM _i is obtained by comparing the image feature amount between the reference image and the evaluation target image Q _i .

In step S33, the similarity evaluation unit 53 extracts a similarity _equal to or greater than a predetermined threshold from the obtained similarities SIM ₁ to SIM _n , and sets the evaluation target image corresponding to the extracted similarity to the similarity of the reference image. Extract as an image. When there are a plurality of similarities equal to or higher than a predetermined threshold, the evaluation target image corresponding to the maximum similarity can be specified as a similar image of the reference image. An evaluation target image specified and extracted as a similar image of the reference image is referred to as a specific similar image (extraction target image).

In step S34, the object position data acquisition unit 54 uses the position data associated with the specific similar image as the object position data (x _T , y _T , z _T ), and the position data (x ₁ , y ₁ , z _1). ) To (x _n , y _n , z _n ) (in other words, extracted from the position data group 341). Thus, for example, if the degree of similarity SIM ₂ is the evaluation object image Q ₂ by being extracted as above similarity predetermined threshold is extracted as a specific similar image, data relating to the position associated with the evaluation object image Q _2, i.e. Then, position data (x ₂ , y ₂ , z ₂ ) representing the location of the object in the evaluation object image Q ₂ is extracted as object position data (x _T , y _T , z _T ).

Returning to the description of FIG. The object position data (x _T , y _T , z _T ) determined by the similar image search process of FIG. 9 is downloaded (received) by the receiving unit 20 in step S16 following step S15. Download refers to receiving arbitrary information from the server device SV via the network NET.

Thereafter, in step S17, the positioning data correction unit 33 performs the original positioning based on the object position data (x _T , y _T , z _T ), the distance image as the distance data, and the shooting direction data (θ, φ). Positioning data correction processing for correcting data (x _M , y _M , z _M ) is performed. In the positioning data correction process in step S17, the object position data (x _T , y _T , z _T ), the distance image, and the shooting direction data (θ, φ) obtained in steps S16, S13, and S12 can be used. .

The positioning data correction process will be described with reference to FIG. Now, as shown in FIG. 11, it is assumed that the object SUB exists at a position 10 m east from the imaging device 1 (thus θ = 270 °), and as a result, in the distance image 320, the object SUB Assume that the subject distance of the center of gravity SUB _P is 10 m. In the example of FIG. 11, it is assumed that φ = 0 °. In this case, the positioning data correction unit 33 regards the object position data (x _T , y _T , z _T ) as position data representing the exact location of the object SUB, and (x _C , y _C , z _C). ) = (X _T −10, y _T , z _T ), the corrected positioning data (x _C , y _C , z _C ) is obtained. That is, the positioning data correction unit 33 converts the original positioning data (x _M , y _M , z _M ) by the positioning processing unit 12 into (x _C , y _C , z _C ) = (x _T −10, y _T , z _T )). The unit of coordinate values x, y, and z is assumed to be m (meter).

As shown in FIG. 12, if θ = 270 ° and 0 ° <φ <90 ° and the subject distance of the center of gravity SUB _P of the object SUB is 10 m in the distance image 320, the positioning data correction unit 33 regards the object position data (x _T , y _T , z _T ) as position data representing the exact location of the object SUB, and (x _C , y _C , z _C ) = (x _T −10 -Corrected positioning data (x _C , y _C , z _C ) is obtained according to cos φ, y _T , z _T −10 · sin φ). That is, the positioning data correction unit 33 converts the original positioning data (x _M , y _M , z _M ) by the positioning processing unit 12 into (x _C , y _C , z _C ) = (x _T −10 · cos φ, y _T , Z _T −10 · sinφ).

After the positioning data correction process, in step S18, the main control unit 21 may correct the positioning data in the image file from (x _M , y _M , z _M ) to (x _C , y _C , z _C ). In addition, the moving route recorded in the log route recording process can be corrected. FIGS. 13A and 13B are correction image diagrams of positioning data. In step S18, the correction information may be displayed on the display unit 17 while performing through display. The correction information is, for example, corrected positioning data, an error between original positioning data and corrected positioning data, or an accurate distance from the imaging apparatus 1 to the target SUB based on the corrected positioning data and target position data.

If the positioning processing unit 12 is used, the current location of the user and the imaging device 1 can be grasped, but the position detection accuracy is limited. If the correction method according to the present embodiment is used, the position detection error can be accurately corrected not only from the image obtained in response to the shutter operation but also from the image obtained for the through display. It becomes possible to know the current location of Since there are images of various objects on the network, even if the object for the imaging apparatus 1 is not a famous object (such as a building of a tourist attraction) (for example, an exhibition hall, a shopping mall, Positioning data can be corrected even if it is an anonymous building or an alley store. Further, by setting a distance image as a reference image and performing a similar image search process based on the distance image, the object recognition is performed when the shooting environment of the imaging apparatus 1 is dark or the object is confused with a background of similar colors. Even under difficult circumstances, it is possible to accurately determine the object, and as a result, it is possible to correct the positioning data accurately.

However, step S14 may be replaced with step S14 'in FIG. That is, in step S14 (step S14 '), the main control unit 21 may set the normal captured image acquired in step S13 as a reference image. When the normal captured image is a reference image, each evaluation target image is a color image of the corresponding target object (represents the intensity and color of light incident on the image sensor of the imaging device that captures the target object from the target object). 2D image) (the same applies to other examples described later). However, when the normal captured image is set as the reference image, it becomes difficult to accurately determine the object under the above-described circumstances (for example, when the imaging environment of the imaging apparatus 1 is dark), and as a result, the distance image is converted into the reference image. The correction accuracy of positioning data may be lower than the case of using as the positioning data.

Further, the display of the correction information in step S18 may include display of warning information (hereinafter referred to as warning display). In step S18, the main control unit 21 can obtain a distance _ΔERR (distance in real space) between the position indicated by the original positioning data and the position indicated by the corrected positioning data. The distance _ΔERR represents a positioning error by the positioning processing unit 12 and corresponds to a correction amount to the original positioning data by the positioning data correcting unit 33. The main control unit 21 may cause the display unit 17 to display warning information (characters, figures, icons, etc.) indicating that the positioning error is large when the distance _ΔERR is greater than or _equal to a predetermined threshold. The warning display is a type of warning notification that suggests a large positioning error, and the warning notification may be any notification that appeals to the human senses (particularly visual, auditory, and tactile). That is, the main control unit 21 outputs video using the display unit 17, audio output using a speaker (not shown) in the imaging device 1, or the imaging device 1 when the distance _ΔERR is greater than or _equal to a predetermined threshold. The user may be notified that the positioning error is large by vibration or the like (the same applies to other embodiments described later). “Large positioning error” in the warning notification means that the positioning error (that is, the distance Δ _ERR ) is larger than a predetermined allowable error.

<< Second Example >>
A second embodiment of the positioning system will be described. FIG. 15 is an operation flowchart of the positioning system according to the second embodiment. First, in step S51, the imaging apparatus 1 and the correction application software are started by turning on the imaging apparatus 1, and thereafter, through display is continuously performed, while the processes in steps S52 to S59 are sequentially performed. The

In step S52, the positioning processing unit 12 acquires original positioning data (x _M , y _M , z _M ). In subsequent step S53, the transmission unit 19 uploads (transmits) the original positioning data (x _M , y _M , z _M ) acquired in step S52 under the control of the main control unit 21. Based on the uploaded original positioning data (x _M , y _M , z _M ), the image restriction selection unit 52 (see FIG. 8) provided in the server device SV follows the method described in the first embodiment, and the image group In 340, the evaluation target images to be supplied to the similarity evaluation unit 53 are narrowed down. That is, the selection unit 52 supplies the similarity evaluation unit 53 with only the image of the object having a location near the position (x _M , y _M , z _M ) in the image group 340. A part of 340 is selected and selected. As in the first embodiment, it is assumed that the plurality of evaluation target images selected here are evaluation target images Q ₁ to Q _n . Note that the shooting direction data may also be given to the selection unit 52 by uploading, and the selection may be performed using the shooting direction data (further narrowing may be performed using the shooting direction data).

The server device SV according to the second embodiment transmits the selected plurality of evaluation target images to the imaging device 1 via the network NET together with the plurality of position data corresponding to the selected plurality of evaluation target images. Conversely, in step S54, the receiving unit 20 uses the evaluation target images Q ₁ to Q _n for the objects around the position (x _M , y _M , z _M ) as the corresponding position data (x ₁ , y ₁ , z ₁ ) to (x _n , y _n , z _n ) are downloaded (received). Thereafter, the normal captured image 310 and the distance image 320 are acquired by the acquisition process in step S55 (see FIGS. 7A and 7B). The acquisition process in step S55 is the same as that in step S13 (see FIG. 6). However, the acquisition in step S55 may be performed between the processes in steps S51 to S54.

In subsequent step S56, the positioning processing unit 12 and the imaging direction detection unit 13 acquire the latest original positioning data (x _M , y _M , z _M ) and imaging direction data (θ, φ). At this time, the main control unit 21 may acquire a focal length to be included in the additional data (see FIG. 5). In the second embodiment, the object position data (x _T , y _T , z _T ) is determined by executing the similar image search process in step S57 on the imaging device 1 side. In order to realize this, as shown in FIG. 16, the main control unit 21 may be provided with a similarity evaluation unit 53 and an object position data acquisition unit 54 (see also FIG. 8). A part including the similarity evaluation unit 53 and the object position data acquisition unit 54 included in the main control unit 21 may be called an arithmetic processing unit 34. The evaluation target images Q ₁ to Q _n and the position data (x ₁ , y ₁ , z ₁ ) to (x _n , y _n , z _n ) downloaded in step S54 are supplied to the arithmetic processing unit 34 for calculation. The processing unit 34 performs steps S32 to S34 in FIG. Thereby, in step S57 and the arithmetic processing unit 34, the similarity between the reference image and each evaluation target image is evaluated (that is, the similarity indicating the comparison result is derived by comparing the reference image with each evaluation target image). A specific similar image (extraction target image) that is a similar image of the reference image is specified and extracted from the evaluation target images Q ₁ to Q _n , and the position data associated with the specific similar image is the target position data (X _T , y _T , z _T ) is extracted from the position data (x ₁ , y ₁ , z ₁ ) to (x _n , y _n , z _n ).

The main control unit 21 desirably sets the distance image acquired in step S55 as the reference image in step S57. In this case, as described in the first embodiment, each evaluation object image is also a distance image for the corresponding object. However, the normal captured image acquired in step S55 may be set as the reference image in step S57.

In step S58 following step S57, the positioning data correction unit 33 performs positioning data correction processing. The positioning data correction process in step S58 is the same as that in step S17 in FIG. However, in the positioning data correction process in step S58, the object position data (x _T , y _T , z _T ), the distance image, and the shooting direction data (θ, φ) obtained in steps S57, S55, and S56 are used. Can do. Thereafter, the process of step S59 is performed. The process of step S59 is the same as that of step S18 of FIG.

The same effect as the first embodiment can be obtained by the second embodiment. In the second embodiment, a similar image can be searched on the imaging device 1 side without burdening the server device SV and without depending on the search capability of the server device SV. Is also expected.

In the imaging apparatus 1, a normal captured image and a distance image obtained by shooting, and an arbitrary image (including an evaluation target image) acquired via the network NET, original positioning data corresponding to them, corrected positioning, and the like. It may be stored and recorded in the database 16 together with the data or the position data (the same applies to the first embodiment described above and other embodiments described later). In addition, when a user operation for registering the shooting position of the normal shooting image and the distance image as a favorite place is performed on the operation unit 18, the distance image acquired by the imaging device 1 and the corrected positioning data are associated with each other and stored in the database 16. You may make it record (it is the same also in the above-mentioned 1st Example and the other Example mentioned later). Information stored and recorded in the database 16 (including images and data) may be used for subsequent positioning data correction processing.

<< Third Example >>
A third embodiment of the positioning system will be described. FIG. 17 is an operation flowchart of the positioning system according to the third embodiment. First, in step S71, the image pickup apparatus 1 and the correction application software are activated by turning on the image pickup apparatus 1. Thereafter, through display is continuously executed, while the processes in steps S72 to S80 are sequentially executed. The

In step S72, the positioning processing unit 12 acquires original positioning data (x _M , y _M , z _M ). In subsequent step S73, the transmission unit 19 uploads (transmits) the original positioning data (x _M , y _M , z _M ) acquired in step S72 under the control of the main control unit 21.

Based on the uploaded original positioning data (x _M , y _M , z _M ), the server device SV returns a recommended shooting spot having a location near the position (x _M , y _M , z _M ) to the imaging device 1. It is recommended to the user to shoot at the recommended shooting spot.

Specifically, the evaluation for the server device SV is based on uploaded original positioning data _{_{_{(x M, y M, z}}} M) , position _{_{_{(x M, y M, z}}} M) object having a location in the vicinity of A target image is extracted from the database 51. The evaluation target image extracted here is called a recommended shooting image. The position data stored in the database 51 and associated with the recommended shooting image is referred to as recommended shooting position data.

The recommended shooting image and the recommended shooting position data are downloaded (received) together with the corresponding navigation information by the receiving unit 20 in step S74. The recommended shooting image is an image for which shooting is recommended for correcting the positioning data, and an image that the user is likely to desire shooting. Therefore, it is desirable that the target object in the recommended photographing image is an object as famous as possible (for example, a temple at a tourist attraction). In the third example, it is assumed that each evaluation target image is composed of a color image for the target object and a distance image for the target object. For the sake of specific description, in the following description, it is assumed that the recommended shooting image to be downloaded includes the recommended shooting color image 410 and the recommended shooting distance image 420 shown in FIGS.

The navigation information corresponding to the recommended shooting image is information for guiding the user to a position suitable for shooting the recommended shooting image, and includes, for example, map information. The user moves to a position suitable for shooting a recommended shooting image according to the navigation information while referring to the recommended shooting color image 410 downloaded and displayed on the display unit 17, and then presses the shutter button 18 a. As a result, in step S75, the normal captured image 310 and the distance image 320 are acquired by the imaging device 1 (see FIGS. 7A and 7B). In subsequent step S76, the positioning processing unit 12 and the imaging direction detection unit 13 acquire the latest original positioning data (x _M , y _M , z _M ) and imaging direction data (θ, φ). At this time, the main control unit 21 may acquire a focal length to be included in the additional data (see FIG. 5).

Also in the third embodiment, as shown in FIG. 16, a similarity evaluation unit 53 and an object position data acquisition unit 54 are provided in the main control unit 21. In step S77, the similarity evaluation unit 53 of the main control unit 21 sets the distance image 320 as a reference image, and then compares the reference image and the recommended shooting distance image 420 to compare the reference image and the recommended shooting distance image. The similarity between 420 is evaluated. Alternatively, in step S77, the similarity evaluation unit 53 of the main control unit 21 sets the normal captured image 310 as a reference image, and then compares the reference image and the recommended shooting color image 410 to compare the reference image and the recommended shooting color. The similarity between the images 410 may be evaluated. The method for deriving the similarity between two images is as described in the first embodiment. Then, the object position data acquisition unit 54 of the main control unit 21 confirms whether the similarity corresponding to the above comparison result is equal to or greater than a predetermined threshold, whereby the

image

410 or 420 is a similar image of the reference image. Check if it corresponds to.

The object position data acquisition unit 54 of the control unit 21 determines that the

image

410 or 420 is a similar image of the reference image in step S78 when the similarity (similarity in step S77) is equal to or greater than a predetermined threshold. and extracts the photographing recommended position data from the received content of the reception unit 20 sets the photographing recommended position data object position data _{_{(x T, y T, z}} T) to. As is clear from the above description, the recommended shooting position data set as the object position data (x _T , y _T , z _T ) is data representing the exact location of the target object in the recommended shooting image. If the similarity in step S77 is not greater than or equal to a predetermined threshold value, the process may return to step S75 to prompt re-shooting of the normal shot image and the distance image.

In step S79 following step S78, the positioning data correction unit 33 performs positioning data correction processing. The positioning data correction process in step S79 is the same as that in step S17 in FIG. However, in the positioning data correction process in step S79, the object position data (x _T , y _T , z _T ), distance image, and shooting direction data (θ, φ) set or acquired in steps S78, S75, and S76. Can be used. Thereafter, the process of step S80 is performed. The process of step S80 is the same as that of step S18 of FIG.

As described above, in the third embodiment, in the flow for the convenience of the user to present and guide the recommended shooting spots (such as shooting best points recommended by professional photographers) in tourist spots, Correction is possible. In addition, when an image uploaded by a friend of the user of the imaging apparatus 1 is downloaded as a recommended shooting image, it is possible to perform shooting at the same location as the shooting location of the friend. Then, by uploading an image captured by the imaging device 1 to an image sharing site (for example, Flickr (registered trademark), Facebook (registered trademark), Twitter (registered trademark)) on the network NET, the user of the imaging device 1 and It is also possible to share image information between the user's friends in real time.

Incidentally, in the server apparatus SV, the original positioning data _{_{_{(x M, y M, z}}} M) is 2 or more captured recommended image based on may be extracted. When two or more recommended shooting images are extracted, two or more sets of “recommended shooting images, recommended shooting position data, and navigation information” are downloaded by the receiving unit 20, and the imaging apparatus 1 has a plurality of sets. The user is prompted to select one of a plurality of recommended shooting color images after displaying the recommended shooting color image on the display unit 17. In response to this, the user selects one shooting recommended color image from a plurality of sets of shooting recommended color images. After this selection, the movement of the user and the processing of the imaging device 1 after step S74 are performed as described above.

<< 4th Example >>
A fourth embodiment of the positioning system will be described. In the fourth embodiment, the image pickup unit 11 is provided with a plurality of sets of image pickup devices, optical systems, and ranging light sources. For example, as shown in FIG. 19, the imaging unit 11, provided with an image sensor IS ₁ and optical system and the distance-measuring light source LT ₁ for image sensor IS _1, the image pickup element IS ₂ and optical system for the imaging element IS ₂ and providing a distance measuring light source LT _2. Although it is possible to generate a distance image even in the case of a single eye (that is, even when the number of imaging elements is 1), as is well known, distance data of a portion affected by occlusion cannot be generated with a single eye. . If compound eyes are used, the influence of occlusion can be suppressed.

Each of the image sensors IS ₁ and IS ₂ is the same as the image sensor IS described above. Each of the ranging light sources LT ₁ and LT ₂ may be the same as the above-described ranging light source LT. The imaging elements IS ₁ and IS ₂ are arranged at different positions, and can generate images having parallax with each other. However, a part of the imaging regions of the image sensors IS ₁ and IS ₂ overlap each other. Image sensor IS ₁ and IS _2, respectively, function as the left and right eyes of the image pickup apparatus 1. The distance measuring light source LT ₁ irradiates light on the subject in the imaging region of the image sensor IS ₁ , and the distance measuring light source LT ₂ irradiates light on the subject in the imaging region of the image sensor IS ₂ . Each distance measuring light source may be formed of a plurality of light emitting elements.

The distance measuring light sources LT ₁ and LT ₂ may irradiate light having a plurality of wavelengths to the subject in the imaging regions of the imaging elements IS ₁ and IS ₂ . More specifically, for example, each of the ranging light sources LT ₁ and LT ₂ is formed by an LED (Light Emitting Diode) that emits near-infrared light, and a plurality of sine waves having different wavelengths (for example, several 10) are formed. The emitted light of each LED may be modulated with a sine wave of (1) and projected onto the subject. Reflected light from the subject with respect to the light emitted from each LED is received by the imaging elements IS ₁ and IS ₂ . The distance data generation unit 32 obtains the phase difference between the emitted light and the reflected light for each pixel based on the output image signal of the image sensor IS ₁ , and calculates the subject distance from the phase difference for each pixel. A left eye distance image based on the output image signal of IS ₁ can be generated. Similarly, the distance data generation unit 32 obtains the phase difference between the emitted light and reflected light for each pixel based on an output image signal from the image sensor IS _2, by obtaining the object distance from the phase difference for each pixel, A right eye distance image based on the output image signal of the image sensor IS ₂ can be generated. Then, the distance data generation unit 32 can generate a final distance image in which the influence of occlusion is suppressed by integrating the left eye distance image and the right eye distance image. The final distance image may be a distance image (for example, the distance image 320 in FIG. 7B) to be generated by the distance data generation unit 32 in the first to third embodiments described above or other embodiments described later. .

The image processing unit 31 of FIG. 1, the output image signal from the image sensor IS ₁ can generate a left-eye normal photographed image, it is possible to generate a right-eye normal photographed image from the output image signal from the image sensor IS ₂ . In the process of generating these images, noise removal processing using a median filter or the like may be performed. The left-eye or right-eye normal captured image is a normal captured image (for example, the normal captured image 310 of FIG. 7A) to be generated by the image processing unit 31 in the first to third embodiments described above or other embodiments described later. ). The image processing unit 31 obtains the center-of-gravity coordinates of the object SUB in each of the left eye and right eye normal captured images. The main control unit 21 (distance data generation unit 32) can derive an accurate distance between the imaging device 1 and the object SUB from each barycentric coordinate and the distance data obtained by the main control unit 21, and performs positioning using the derived distance. Data correction processing can be performed.

As described above, the influence of occlusion can be suppressed by performing distance measurement using a plurality of image sensors. Also, when looking at only the phase difference for a single wavelength, there may be multiple distances that give the same phase (as a result, a large error may occur in distance measurement), but a modulated wave with multiple wavelengths should be projected. Thus, the accuracy of distance measurement can be improved. If the phase difference is continuously measured by the image sensor, distance data can be generated in real time.

Although the example in which the image pickup unit 11 is provided with two sets of the image pickup device, the optical system, and the distance measuring light source is described above, the image pickup unit 11 is provided with three sets or more of the image pickup element, the optical system, and the distance measuring light source. Also good.

Further, when the imaging unit 11 is provided with two or more imaging elements having parallax, it is possible to omit the ranging light source from the imaging unit 11. In this case, the distance data generation unit 32 may generate distance data using the principle of triangulation based on output image signals of two or more image sensors.

<< 5th Example >>
A fifth embodiment of the positioning system will be described. The communication unit including the transmission unit 19 and the reception unit 20 may be connected to the network NET through an intermediary device. FIG. 20 is a schematic block diagram showing the overall configuration of the positioning system according to the fifth embodiment, and the positioning system of FIG. 20 is formed by adding an intermediary device CC to the positioning system of FIG. The mediation device CC is a device that has a communication function and can be connected to the network NET, and is, for example, a mobile phone (including a so-called smartphone), an information terminal, or a personal computer. According to the present embodiment, even when the imaging device 1 is a device that cannot always connect to the network NET (for example, a Wi-Fi (registered trademark) authentication device), the correction of the positioning data, etc. via the intermediary device. Is possible.

Although it has been described above that any image (including the image to be evaluated) and position data acquired via the network NET may be accumulated and recorded in the database 16, in the fifth embodiment, the database 16 May be provided in the intermediary device CC. Thereby, the data storage load in the imaging device 1 is reduced.

<< Sixth Example >>
A sixth embodiment will be described. In the sixth embodiment, it is assumed that a specific object capable of transmitting its own position information to the imaging device 1 is located around the imaging device 1. FIG. 21 shows an access point 500 (hereinafter referred to as AP 500) as an example of a specific object. The imaging device 1 is a Wi-Fi (registered trademark) authentication device, for example, and can be connected to the network NET via the AP 500.

The AP 500 has a memory that stores AP position data (x _T2 , y _T2 , z _T2 ) representing the location of the AP 500. In a state where communication between the imaging apparatus 1 and the AP 500 is possible, the AP 500 can transmit AP position data (x _T2 , y _T2 , z _T2 ) to the imaging apparatus 1, and the receiving unit 20 of the imaging apparatus 1 can transmit the AP position data. Data can be received. When receiving the AP position data, the imaging apparatus 1 can notify the user that the AP 500 should be photographed using the display unit 17 or the like. In response to this, the user operates the imaging device 1 to cause the imaging device 1 to photograph the AP 500 as an object. As a result, a captured image of the AP 500 as an object is obtained by the imaging device 1. The captured image obtained here includes a distance image in the imaging region including the AP 500.

The positioning data correction unit 33 captures the object position data as the AP position data (x _T2 , y _T2 , z _T2 ), the distance image obtained by photographing the AP 500 as the object, and at the time of photographing the distance image. Positioning data correction processing for correcting the original positioning data (x _M , y _M , z _M ) can be performed based on the shooting direction data. The specific contents of the positioning data correction process are as described in the above embodiment.

Also in the sixth embodiment, the same effect as in the first embodiment can be obtained. In the sixth embodiment, the positioning data correction can be completed in a short time without requiring the server device SV. In the positioning data correction process, in order to make the imaging apparatus 1 accurately recognize which object on the distance image is the AP 500, the AP 500 is information indicating its own shape (for example, the image of the AP 500 itself) or the AP 500. May be transmitted to the imaging apparatus 1 (for example, an identification mark unique to the AP 500 that can be observed by the imaging apparatus 1).

<< Deformation, etc. >>
The embodiment of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims. The above embodiment is merely an example of the embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the above embodiment. The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As annotations applicable to the above-described embodiment, annotations 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The shooting direction data detected and recorded by the shooting direction detection unit 13 and the shooting direction recording unit 14 may include a roll angle. The roll angle is an angle in the roll direction of the imaging device 1 and is detected using a magnetic sensor or a gyro sensor. The rotation angle in the motion of rotating the housing of the imaging device 1 around the optical axis in a state where the optical axis of the imaging unit 11 is unchanged is the roll angle. In a state in which the horizontal direction in the captured image (normally captured image, distance image) of the imaging device 1 is parallel to the horizontal plane, the roll angle is 0 degrees, and the horizontal direction of the captured image increases from the horizontal plane as the absolute value of the roll angle increases. Tilt. In each of the above-described embodiments including the first to third embodiments, the main control unit 21 corrects the distance image or the normal captured image with the roll angle (the horizontal image of the distance image or the normal captured image with respect to the horizontal plane). The image whose inclination is corrected) can be set as the reference image. Thereby, it is possible to realize similar image search processing, similarity evaluation processing, and positioning data correction processing that do not depend on the roll angle.

[Note 2]
The posture angle φ can be deleted from the shooting direction data. In addition, the original positioning data, the corrected positioning data, and the arbitrary position data (including the object position data) may not include the Z-axis component that is an altitude component. When the Z-axis component is not included in the original positioning data, the corrected positioning data, and the arbitrary position data, the posture angle φ may be deleted from the shooting direction data.

[Note 3]
Each of the imaging device 1 and the server device SV can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 or the server device SV is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 12 Positioning process part 13 Shooting direction detection part 19 Transmission part 20 Reception part 32 Distance data generation part 33 Positioning data correction part 34 Operation processing part 52 Image limitation selection part 53 Similarity evaluation part 54 Object position data Acquisition unit IS, IS ₁ , IS ₂ Image sensor LT, LT ₁ , LT ₂ Distance measuring light source NET network SV, SV _A server equipment

Claims

An imaging unit having an imaging element that outputs an image signal of a subject;
A distance data generation unit that generates distance data representing a distance between the electronic device and the subject using the output of the imaging element;
A shooting direction detection unit that generates shooting direction data by detecting a shooting direction of the imaging unit;
A positioning processing unit that performs positioning based on signals from satellites and generates original positioning data;
A transmission unit that transmits a reference image based on the output of the image sensor to an external device;
A receiving unit for receiving position data based on the reference image from the external device;
An electronic apparatus comprising: a positioning data correction unit that corrects the original positioning data based on reception position data, the distance data, and the shooting direction data.
Data relating to the position associated with the target image corresponding to the reference image is received as the position data by the receiving unit,
The electronic device according to claim 1, wherein the data relating to the position associated with the target image includes location data of an object included in the target image.
The transmitting unit transmits the reference image and the original positioning data to an external device,
The electronic apparatus according to claim 1, wherein the reception position data is position data based on the reference image and the original positioning data.
An imaging unit having an imaging element that outputs an image signal of a subject;
A distance data generation unit that generates distance data representing a distance between the electronic device and the subject using the output of the imaging element;
A shooting direction detection unit that generates shooting direction data by detecting a shooting direction of the imaging unit;
A positioning processing unit that performs positioning based on signals from satellites and generates original positioning data;
A transmission unit for transmitting the original positioning data to an external device;
A receiving unit for receiving a target image and position data based on the original positioning data from the external device;
An arithmetic processing unit that compares the target image with a reference image based on the output of the image sensor and extracts the position data from the received content of the receiving unit according to a comparison result;
An electronic apparatus comprising: a positioning data correction unit that corrects the original positioning data based on the extracted position data, the distance data, and the shooting direction data.
The electronic device according to claim 5, wherein the position data includes location data of an object included in the target image.
The receiving unit receives a target image group and a position data group based on the original positioning data from the external device,
The arithmetic processing unit extracts an image corresponding to the reference image from the target image group as the target image, and extracts the position data associated with the target image from the position data group. The electronic device according to claim 4 or 5.
7. The electronic apparatus according to claim 1, wherein a distance image based on the distance data is used as the reference image.
The imaging unit includes a ranging light source that irradiates light to the subject,
8. The electronic device according to claim 1, wherein the distance data is generated using the distance measuring light source.
The imaging unit has a plurality of imaging elements as the imaging element and a plurality of ranging light sources corresponding to the plurality of imaging elements,
The plurality of ranging light sources irradiate the subject with light having a plurality of wavelengths,
8. The electronic apparatus according to claim 1, wherein the distance data generation unit generates the distance data using an output of each image sensor depending on reflected light from the subject. .
10. The electronic device according to claim 1, wherein warning notification is made according to an amount of correction to the original positioning data by the positioning data correction unit.