JP2023115289A - Detection device, information processing device, and system - Google Patents

Abstract

To provide an imaging device capable of reducing the processing load.SOLUTION: An imaging device includes: a main body part; a detection unit including an imaging unit that captures an image of a target object, and a ranging unit that detects distance from each point on a surface of the target object; an information calculation unit that is provided in the main body part and calculates at least one of shape information and texture information of the target object using detection results of the detection unit; and a communication unit that transmits calculation results of the information calculation unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to a detection device, an information processing device, and a system.

A technique has been proposed in which an object is imaged by a plurality of imaging devices, the obtained images are input to a computer, and a three-dimensional shape of the object is acquired (see, for example, Patent Document 1).

JP 2010-134546 A

For example, when generating a three-dimensional shape of an object from a plurality of images captured by an imaging device, it is necessary to transfer all images from the imaging device to a computer and process those images on the computer. . In this case, since all captured images are transferred from the imaging device to the computer, communication (eg, communication processing, communication data amount, etc.) between the imaging device and the computer may increase. SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus, an information processing apparatus, and an imaging system that can reduce the communication load.

According to the aspect of the present invention, a detection unit that detects an object within a visual field range, an information generation unit that generates shape information of the object based on the detection result of the detection unit, and a request signal from an external device is received. and a communication unit for transmitting shape information to an external device, the communication unit receiving a request signal specifying a range narrower than the viewing range, and the information generating unit determining the shape of the range specified by the request signal. A detection device is provided that generates information and the communication unit transmits the shape information of the area.

According to an aspect of the present invention, a first detector that detects a captured image of an object and a distance from the object and generates first shape information that indicates the shape of a portion, but not all, of the surface of the object; An information processing device that communicates with a second detection device that detects a captured image of an object and a distance from the object, and generates second shape information that indicates the shape of a part, but not all, of the surface of the object , a receiving unit for receiving additional information indicating the positional relationship between the first shape information and the second shape information; a transmitting unit that transmits to the first detection device and transmits to the second detection device a request signal that requests the second shape information in a range determined based on the additional information of the second shape information, to the second detection device; provided.

According to an aspect of the invention, a first detection device detects an object from a first position and generates first shape information of the object; A second detection device that detects from the position of the object and generates second shape information of the object, and is capable of communicating with the first detection device and the second detection device, and integrates the first shape information and the second shape information An information processing device for generating integrated shape information is provided, and the information processing device transmits information indicating a portion of the first shape information necessary for generating the integrated shape information to the first detection device, and generates the integrated shape information. A system is provided for transmitting information to the second detection device indicating the portion of the second shape information required to perform the detection.

According to the present invention, it is possible to provide an imaging device, an information processing device, and an imaging system that can reduce communication load.

1 is a diagram showing an imaging system according to a first embodiment; FIG. 1 is a block diagram showing an imaging system according to a first embodiment; FIG. It is a figure which shows the detection part which concerns on 1st Embodiment. It is a figure which shows operation|movement of the imaging system which concerns on 1st Embodiment. FIG. 11 is a block diagram showing an imaging system according to a second embodiment; FIG. It is a figure which shows operation|movement of the imaging system which concerns on 2nd Embodiment. FIG. 12 is a diagram showing an imaging system according to a third embodiment; FIG. FIG. 11 is a block diagram showing an imaging system according to a third embodiment; FIG. It is a figure which shows operation|movement of the imaging device which concerns on 3rd Embodiment. It is a figure which shows the operation|movement of the imaging system which concerns on 4th Embodiment. FIG. 12 is a block diagram showing an imaging system according to a fifth embodiment; FIG. It is a figure which shows operation|movement of the imaging system which concerns on 5th Embodiment. FIG. 11 is a block diagram showing an imaging system according to a sixth embodiment; FIG. FIG. 12 is a block diagram showing an imaging system according to a seventh embodiment; FIG. It is a figure which shows the operation|movement of the imaging system which concerns on 7th Embodiment. It is a figure which shows operation|movement of the imaging system which concerns on 8th Embodiment. FIG. 22 is a diagram showing the operation of the imaging system according to the ninth embodiment;

[First embodiment]
A first embodiment will be described. FIG. 1 is a diagram showing an imaging system 1 according to this embodiment. For example, the imaging system 1 includes an imaging device 2 , an information processing device 3 , an input device 4 and a display device 5 .

The imaging device 2 captures an image of the object OB illuminated by, for example, light from a lighting device, light from a room light, or natural light, and captures an image of the object OB (eg, visible light image, infrared light image, etc.). image) data. Further, the imaging device 2 detects the distance from the imaging device 2 to each point on the surface of the object OB, and acquires the depth information of the object OB. The depth information includes, for example, information that associates the position of a point on the surface of the object OB with the distance (depth) from this point to the imaging device 2 . The depth information is, for example, information indicating a depth distribution (eg, depth map) in the object OB.

The imaging device 2 uses the data of the captured image captured by the imaging unit 15 and the depth information detected by the distance measuring unit (distance detection unit) 16 to perform arithmetic processing of information regarding the object OB. The imaging device 2 models at least a part of the object OB by arithmetic processing, and calculates model information (model data). For example, the imaging device 2 performs computer graphics processing (CG processing) on at least a portion of the object OB by arithmetic processing, and calculates model information (eg, CG model data). The model information includes, for example, at least one of shape information indicating the three-dimensional shape of the object OB and texture information indicating the surface pattern of the object OB. For example, the model information includes three-dimensional point coordinates, related information of the point coordinates, surface texture information defined by the point coordinates and related information, image space such as illumination conditions and light source information of the entire image. information, and at least one of polygon data as shape information.

For example, the object OB in FIG. 1 includes a prismatic member and a soccer ball, and the shape information includes information on the surface (eg, plane) of the prismatic member and information on the surface (eg, spherical) of the soccer ball. include. Texture information includes, for example, characters and figures (eg, black pentagons, white hexagons) on the surface of a soccer ball, patterns, information that defines unevenness, specific images, and colors (eg, chromatic and achromatic colors). ).

The imaging device 2 calculates, for example, model information of a portion (visual field region) of the object OB that is within the field of view of the imaging device 2 . This model information is, for example, information of a model (hereinafter referred to as a partial model) representing a part of a model representing the entire circumference of the object OB (hereinafter referred to as an all-around model). The imaging device 2 can supply at least part of the information (eg, model information) calculated by arithmetic processing to an external device. For example, the imaging device 2 supplies at least part of the model information to the information processing device 3 .

Also, for example, the imaging device 2 can output at least part of the model information to a digital device capable of inputting/outputting digital information such as a bar code or a two-dimensional code. Such digital devices are capable of displaying or printing digital information, including at least part of the model information, on a display, paper, or the like. A reader device having a reader unit (eg, an optical reader) capable of reading displayed or printed digital information can input the digital information into its own storage area or the like via the reader unit. Also, the reader device may further include a rendering processing unit, which will be described later. Note that the imaging system 1 described above may be configured to include the digital device described above and a reader device including a reader section. Further, the imaging device 2 may be configured to include the above-described digital device or reader device.

The information processing device 3 includes, for example, a computer system. The input device 4 includes at least one of, for example, a keyboard, a mouse, a touch panel, a sensor such as an acceleration sensor, a voice input device, and a touch pen, and is connected to the information processing device 3 . The input device 4 receives input of information from a user, for example, and supplies the input information to the information processing device 3 . The display device 5 includes, for example, a liquid crystal display or a touch panel display, and is connected to the information processing device 3 . The display device 5 displays an image based on image data supplied from the information processing device 3, for example.

The information processing device 3 is communicably connected to the imaging device 2 by wire or wirelessly. For example, the information processing device 3 may be connected to the imaging device 2 via a communication cable, or may be connected to the imaging device 2 via an Internet line. Further, the information processing device 3 may be capable of communicating with the imaging device 2 using radio waves or infrared rays.

The information processing device 3 acquires information from the imaging device 2 through communication with the imaging device 2 . The information processing device 3 uses information (eg, model information) acquired from the imaging device 2 to execute rendering processing. For example, the information processing device 3 calculates data of an estimated image of the object OB viewed from this viewpoint, based on the setting information of the viewpoint (imaging direction) input to the input device 4 by the user. The information processing device 3 supplies, for example, the data of the estimated image to the display device 5 and causes the display device 5 to display the estimated image.

The imaging device 2 extracts information used for rendering processing of the information processing device 3, for example, from data of the captured image and depth information. The information extracted by the imaging device 2 is, for example, at least part of the model information. Since the imaging device 2 supplies the extracted information to the information processing device 3, for example, compared to the case where the data of the captured image and the depth information are supplied to the information processing device 3, the imaging device 2 and the information processing device 3 It is possible to reduce the communication load between and the processing load of the information processing device 3 .

Note that the imaging device 2 may be, for example, a portable information terminal, a fixed-point camera, or a camera whose field of view can be changed manually or automatically. The imaging system 1 may include, for example, a moving device that moves the imaging device 2 . This moving device may change the field of view of the imaging device 2 by moving the imaging device 2 . A control unit that controls this moving device may be provided in the imaging system 1 , for example, may be provided in the imaging device 2 , or may be provided in the information processing device 3 . Also, the control unit that controls this moving device may be provided in a device external to the imaging system 1 .

Next, each part of the imaging system 1 will be described. FIG. 2 is a block diagram showing the imaging system 1 according to this embodiment. The imaging device 2 includes a detection section 11 , an information calculation section 12 , a communication section 13 and a main body section 14 . The body part 14 is, for example, a camera body, a case, a housing, or the like. The detection unit 11 , the information calculation unit 12 and the communication unit 13 are provided in the main unit 14 .

The detection unit 11 includes an imaging unit 15 and a distance measurement unit 16 . FIG. 3A is a diagram showing an example of the detection unit 11. As shown in FIG. The imaging unit 15 includes an imaging optical system 17 and an image sensor 18 . The image sensor 18 is, for example, a CMOS image sensor in which a plurality of pixels are two-dimensionally arranged, or a CCD image sensor. The image sensor 18 is housed in the body portion 14 . The image sensor 18 has sensitivity to, for example, a visible light wavelength band (eg, 380 nm or more and 750 nm or less) or an infrared light wavelength band. The imaging result of the image sensor 18 includes, for example, information (eg, RGB data) of gradation value for each color of each pixel. The image sensor 18 outputs, for example, the imaging result (detection result) in the data format of a full-color image. A full-color image is, for example, an image in which each of red (R), green (G), and blue (B) of each pixel is represented by a gradation value (eg, 256 gradations).

The imaging optical system 17 includes, for example, a plurality of lenses, and forms an image of an object plane (eg, object OB) on the image sensor 18 . The imaging optical system 17 is held in, for example, a lens barrel and attached to the main body 14 together with the lens barrel. The imaging optical system 17 and the lens barrel are interchangeable lenses, for example, and can be removed from the main body 14 . The lens barrel may be part of the body portion 14 or may not be detachable from the body portion 14 .

A distance measuring unit 16 detects the distance from each point on the surface of the object OB. The distance measuring unit 16 includes, for example, a ToF (Time of Flight) sensor, and detects distance by the ToF method. Also, for example, the distance measuring unit 16 includes a phase difference sensor and detects distance by a phase difference method. The distance measuring section 16 includes an irradiation section 19 , an imaging optical system 20 , an image sensor 21 and a controller 22 .

The irradiation unit 19 can irradiate the object OB with infrared light. The irradiation unit 19 is controlled by the controller 22 . The controller 22 temporally changes the intensity of the infrared light emitted from the irradiation unit 19 (for example, modulates the amplitude).

The image sensor 21 is, for example, a CMOS image sensor in which a plurality of pixels are two-dimensionally arranged, or a CCD image sensor. The image sensor 21 is housed in the body portion 14 . The imaging optical system 20 includes, for example, a plurality of lenses, and forms an image of an object plane (eg, object OB) on the image sensor 21 . The imaging optical system 20 may be an interchangeable lens, or may be built in the body section 14 . The image sensor 21 has sensitivity to at least the wavelength band of the light emitted by the irradiation unit 19 . For example, the image sensor 21 is controlled by the controller 22 and detects infrared light reflected and scattered by the object OB.

The controller 22 detects the distance from the surface of the object OB to the image sensor 21 using the detection result detected by the image sensor 21 of the distance measuring section 16 . For example, if the depth is different between the first position and the second position on the surface of the object OB, the light reflected and scattered at the first position is more distant from the object OB than the light reflected and scattered at the second position. The flight distance (optical path) to the image sensor 21 is different. Therefore, the light reflected and scattered at the first position of the object OB is incident on the image sensor 21 in a phase different from the phase of the light reflected and scattered at the second position. Since the controller 22 time-modulates the intensity of the light emitted from the irradiation unit 19, the intensity of the light incident on the image sensor 21 changes depending on the phase. The controller 22 calculates the depth, for example, based on the time change of the output of the image sensor 21 . For example, the controller 22 calculates the depth from the distance between the image sensor 21 and the surface of the object OB for each partial area (eg, one pixel, a plurality of pixels) of the image captured by the image sensor 21. The depth (or distance) and the position of the partial area are associated. Thus, the controller 22 calculates, for example, depth information including the depth of each of the partial regions.

Note that the irradiation unit 19 may emit light including visible light, and the image sensor 21 may detect light reflected and scattered by the object OB. For example, the distance measurement unit 16 detects the object OB by the image sensor 21 in the light irradiation state and the light non-irradiation state by the irradiation unit 19, and uses the difference between the detection results in the two states to obtain depth information. may be calculated.

Note that the distance measuring unit 16 may include a projector camera. In this case, the distance measurement unit 16 irradiates the object OB with light having a predetermined intensity distribution (eg, pattern light, structured light, texture) from the irradiation unit 19, while the image sensor 21 detects the object OB. Take an image of the OB. The controller 22 detects the distance between the object OB and the rangefinder 16 using the intensity distribution of the light on the object OB captured by the image sensor 21 . Moreover, the distance measuring unit 16 may include a laser scanner. In this case, the distance measurement unit 16 irradiates the object OB with a laser beam from the irradiation unit 19, scans the object OB with the laser beam, and detects the laser beam reflected by the object OB. and the distance measuring unit 16 is detected.

FIG. 3B is a diagram showing another example of the detection unit 11. As shown in FIG. The distance measurement unit 16 of the detection unit 11 detects the distance between the object OB and the detection unit 11 using detection results obtained by detecting the object OB from a plurality of fields of view. The imaging unit 15 includes an imaging optical system 17a and an image sensor 18a, and an imaging optical system 17b and an image sensor 18b. The imaging optical system 17a and the image sensor 18a image the object OB in the first field of view. The imaging optical system 17b and the image sensor 18b image the object OB in the second field of view. The second field of view differs from the first field of view in at least one of the position of the viewpoint and the direction of the line of sight. The position of the viewpoint is the position where the image of the object is formed, such as the position of the image sensor 18a and the position of the image sensor 18b. The direction of the line of sight is, for example, parallel to the direction of the optical axis of the optical member arranged closest to the object side among the optical members through which the light traveling toward the viewpoint passes. For example, the line-of-sight direction of the first visual field is coaxial with the optical axis on the exit side of the imaging optical system 17a. Also, the direction of the line of sight of the second visual field is coaxial with the optical axis on the exit side of the imaging optical system 17b. The distance measurement unit 16 detects the distance between the object OB and the detection unit 11 by using the image captured by the image sensor 18a and the image captured by the image sensor 18b as parallax images.

Note that the imaging unit 15 captures an image from the first field of view and an image from the second field of view at different times using one image sensor, and the distance measuring unit 16 uses these images to detect the object OB and the detection unit. 11 may be detected.

Note that the distance measuring unit 16 may detect a distance represented by a dimensional value such as meters, or may detect a relative distance represented by a dimensionless value normalized by a reference value. good. Further, the distance measuring section 16 may be, for example, in a unitized form, or may be externally attached to the main body section 14 . Moreover, at least part of the optical system provided in the distance measuring section 16 may be shared with at least part of the optical system provided in the imaging section 15 . For example, in FIG. 3A, the distance measurement unit 16 splits the infrared light from the light that has passed through the imaging optical system 17 with a dichroic mirror or the like, and the image sensor 21 detects the split infrared light. It's okay. Further, in the projector camera, the irradiation unit 19 may project a pattern onto the object OB via the imaging optical system 17 .

Returning to the description of FIG. 2, the information calculation unit 12 includes, for example, a digital signal processor (DSP). The information calculator 12 uses the detection result of the detector 11 to calculate at least one of shape information and texture information of the object OB.

The information calculation unit 12 calculates, as shape information, coordinates of a plurality of points on the surface of the object OB (hereinafter referred to as point cloud data) and surface information including connection information between the points. Surface information is, for example, polygon data, vector data, draw data, and the like. The connection information includes, for example, information for associating points at both ends of lines corresponding to ridges (eg, edges) of the object OB, and information for associating multiple lines corresponding to the contours of the surfaces of the object OB.

First, the information calculation unit 12 calculates point cloud data using the detection result (eg, depth information) of the distance measurement unit 16 (point cloud data processing). For example, the information calculation unit 12 calculates the point cloud data by perspective transformation from the distance image indicated by the depth information to a plane image. In addition, when the fields of view of the imaging unit 15 and the distance measuring unit 16 are different, the information calculating unit 12 performs perspective transformation (projective transformation) on the detection result of the distance measuring unit 16, for example, from the visual field of the imaging unit 15 to the target object. OB may be converted into the result of detection. For example, the information calculation unit 12 performs perspective transformation using parameters that depend on the positional relationship between the field of view of the imaging unit 15 and the field of view of the distance measurement unit 16 (eg, the position of the viewpoint, the direction of the line of sight). good.

Then, the information calculation unit 12, for example, estimates a plane between a point selected from a plurality of points included in the point cloud data and its neighboring points, and converts the point cloud data into a polygon having plane information between the points. Convert to data (surface processing). The information calculation unit 12 converts the point cloud data into polygon data by, for example, an algorithm using the method of least squares. This algorithm may be, for example, an algorithm published in the point cloud processing library.

Next, the information calculation unit 12 calculates texture information by, for example, an inverse rendering technique. The texture information includes, for example, pattern information indicating the pattern of the surface of the object OB, light source information of the light illuminating the object OB, and optical characteristics indicating the optical characteristics (eg, reflectance, scattering ratio) of the surface of the object OB. Contains at least one item of information. The light source information includes, for example, information on at least one of the position of the light source, the direction in which light is emitted from the light source to the object, the wavelength of the light emitted from the light source, and the type of light source.

The information calculation unit 12 calculates light source information using, for example, a model assuming Lambertian reflection, a model including albedo estimation, or the like. For example, the information calculation unit 12 estimates the component derived from the light diffused by the object OB and the component specularly reflected by the object OB among the pixel values of each pixel of the image captured by the imaging unit 15 . Further, the information calculation unit 12 calculates the direction in which light is incident on the object OB from the light source, for example, using the estimation result of the component specularly reflected by the object OB and the shape information. The information calculator 12, for example, estimates the reflection characteristics of the object OB using the calculated light source information and shape information, and calculates optical characteristic information including the estimation result of the reflection characteristics. Further, the information calculation unit 12 calculates pattern information by removing the influence of illumination light from the visible light image data, for example, using the calculated light source information and optical characteristic information.

Here, the imaging device 2 according to the present embodiment further includes a display section 25, an input section 26, a storage section 27, and a control section . The display unit 25 (see FIG. 1) is, for example, a liquid crystal display or a touch panel type display provided in the main unit 14 . The display unit 25 displays, for example, the detection result of the detection unit 11 (eg, visible light image by the imaging unit 15), the calculation result of the information calculation unit 12 (eg, depth map), and at least part of various setting information. .

The input unit 26 is, for example, an operation button provided on the main unit 14, a touch panel provided on the display unit 25, a voice input device for recognizing the voice of the user, or the like. The input unit 26, for example, detects an operation by a user and receives input of information from the user. The input unit 26 transmits input information to the control unit 28 .

The storage unit 27 is, for example, a non-volatile memory such as a USB memory or a memory card, and stores various information. The storage unit 27 may include a storage device built into the imaging device 2 or may include a port to which a storage device releasable from the imaging device 2 can be connected. Further, the information calculation unit 12 of the imaging device 2 according to the present embodiment generates model information by attaching header information (eg, identification information such as a number or code) to information including at least one of shape information and texture information. Then, the communication unit 13 transmits the model information. The header information includes identification information, the position (positional information) of the imaging device 2, the imaging timing by the imaging unit 15, the imaging time by the imaging unit 15, the optical characteristic information of the object OB, and the imaging environment information (eg, light source). information, illumination conditions for the object OB, etc.). In this manner, the information calculation unit 12 of the imaging device 2 can generate model information having header information based on a predetermined data format and cause the communication unit 13 to transmit the model information.

The control unit 28 controls each unit of the imaging device 2 according to a command (control signal) from a user or an external device (eg, the information processing device 3), for example. For example, the control unit 28 causes the detection unit 11 to perform the detection process described above. This detection processing includes, for example, imaging processing by the imaging unit 15 and distance detection processing by the distance measuring unit 16 . The control unit 28 stores at least part of the detection result of the detection unit 11 in the storage unit 27, for example. For example, the control unit 28 causes the information calculation unit 12 to calculate model information. For example, the control unit 28 causes the storage unit 27 to store at least part of the model information calculated by the information calculation unit 12 .

The storage unit 27 stores, for example, model information for each item. For example, the storage unit 27 stores each item of shape information, texture information, light source information, optical characteristic information of the object OB, and pattern information in individual data tables. The control unit 28 causes the display unit 25 to display, for example, an image showing at least part of the information stored in the storage unit 27 . The control unit 28 controls the communication unit 13 to execute information transmission and information reception via the communication unit 13 .

The communication unit 13 includes, for example, at least one of an I/O port such as a USB port and a communication device that performs wireless communication using radio waves or infrared rays. The communication unit 13 is controlled by the control unit 28 to read information stored in the storage unit 27 and transmit the read information to an external device. For example, the communication unit 13 transmits at least part of the calculation result (eg, model information) of the information calculation unit 12 to the information processing device 3 . Also, the communication unit 13 receives information including commands from an external device, for example. The communication unit 13 can store the received information in the storage unit 27 and supply the received information to the control unit 28 . In addition, when the imaging device 2 includes the above-described digital device, the communication unit 13 may transmit at least part of the model information to the digital device. The digital device may then generate digital information based on the received model information and output the digital information to a medium such as paper.

As shown in FIG. 2 , the information processing device 3 includes a communication section 30 , a storage section 31 , a rendering processing section 32 and a control section 33 . The communication unit 30 includes, for example, at least one of a USB port, a network card, and a communication device that performs wireless communication using radio waves or infrared rays. The communication unit 30 can communicate with the communication unit 13 of the imaging device 2 .

The storage unit 31 includes, for example, a removable storage medium such as a USB memory, and a large-capacity storage device such as an external or internal hard disk. The storage unit 31 stores, for example, at least part of data of information received via the communication unit 30, an imaging control program for controlling the imaging device 2, a processing program for executing each process of the information processing device 3, and the like. .

The rendering processing unit 32 includes, for example, a graphics processing unit (GPU). Note that the rendering processing unit 32 may be configured such that the CPU and memory execute each process according to an image processing program. The rendering processing unit 32 performs, for example, at least one of drawing processing, texture mapping processing, and shading processing.

In the drawing process, the rendering processing unit 32 can calculate an estimated image (eg, reconstructed image) of the shape defined in the shape information of the model information viewed from an arbitrary viewpoint. In the following description, a shape indicated by shape information is called a model shape. The rendering processing unit 32 can reconstruct a model shape (eg, estimated image) from model information (eg, shape information) by, for example, drawing processing. The rendering processing unit 32 stores the calculated data of the estimated image in the storage unit 31, for example. Since the imaging device 2 can transmit at least part of the model information to the information processing device 3, the information processing device 3 can reduce the load of rendering processing, for example. Further, for example, the imaging device 2 does not need to transmit all captured images captured by the imaging unit 15 to the information processing device 3, and at least part of the model information (eg, shape information) calculated by the information calculation unit 12 and texture information) to the information processing device 3 . Therefore, the imaging apparatus of this embodiment can reduce the communication load of information necessary for the drawing processing of the rendering processing unit 32 .

Also, in the texture mapping process, the rendering processing unit 32 can calculate an estimated image in which, for example, an image indicated by the texture information of the model information is pasted on the surface of the object on the estimated image. The rendering processing unit 32 can also calculate an estimated image in which a texture different from that of the object OB is applied to the surface of the object on the estimated image.

In the shading process, the rendering processing unit 32 can calculate an estimated image in which, for example, shadows formed by the light source indicated by the light source information of the model information are added to the object on the estimated image. Also, in the shading processing, the rendering processing unit 32 can calculate an estimated image in which a shadow formed by an arbitrary light source is added to an object on the estimated image, for example.

The control unit 33 controls each unit of the information processing device 3, the imaging device 2, the input device 4, and the display device 5, for example. For example, the control unit 33 controls the communication unit 30 to transmit commands (control signals) and setting information to the imaging device 2 . For example, the control unit 33 causes the storage unit 31 to store information received by the communication unit 30 from the imaging device 2 . For example, the control unit 33 controls the rendering processing unit 32 to execute rendering processing.

The control unit 33 controls the imaging device 2 by sending a command (signal) to the imaging device 2 via the communication unit 30, for example. For example, the control unit 33 controls the communication unit 30 and transmits a command (request signal) requesting transmission of predetermined information to the imaging device 2 . The control unit 33 may transmit a command to the imaging device 2 to cause the imaging device 2 to execute each process. For example, the control unit 33 may transmit a command to the imaging device 2 to cause the detection unit 11 of the imaging device 2 to perform detection processing. For example, the control unit 33 may transmit a command to the imaging device 2 to cause the information calculation unit 12 of the imaging device 2 to execute model information calculation processing. Further, the control unit 33 of the information processing device 3 transmits to the imaging device 2 a command to cause the imaging unit 15 of the imaging device 2 to perform imaging of the object OB and a command to set imaging conditions for the object OB. good too.

For example, the communication unit 13 selectively transmits the information calculated by the information calculation unit 12 for each item. For example, the setting information stored in the storage unit 27 includes transmission item information that determines whether or not to transmit each item of model information, and transmission order information that determines the order in which each item of information is transmitted. include. This setting information can be updated by, for example, an operation of the input unit 26, an instruction from the information processing device 3, or the like. For example, the control unit 28 controls the communication unit 13 to transmit the information of the items specified in the transmission item information in the order specified in the transmission order information. Note that the control unit 28 may, for example, control the communication unit 13 to transmit information of items (eg, shape information and texture information) specified in the transmission item information at once based on a predetermined data format. good.

The transmission item information may be set according to, for example, whether or not it is used for rendering processing of the information processing device 3 . For example, in rendering processing, a texture different from that of the object OB may be combined with the shape of the object OB. In this case, the information processing device 3 can execute rendering processing using, for example, the shape information of the object OB without using the texture information of the object OB. The transmission item information is set to, for example, information specifying that the shape information is transmitted but the texture information is not transmitted. In this case, the information calculation unit 12 does not need to calculate texture information.

Further, for example, in rendering processing, an image may be calculated in which the illumination of the object OB is changed. The information processing device 3 can execute rendering processing using, for example, the shape information, pattern information, and optical characteristic information of the target object OB without using the light source information. In this case, the transmission item information is set to, for example, information specifying that shape information, pattern information, and optical characteristic information should be transmitted, and light source information should not be transmitted. Also, for example, the transmission item information is set to information that defines transmission of at least one of shape information, texture information, pattern information, light source information, and optical characteristic information. The imaging device 2 can reduce the communication load when transmitting part of the model information, for example.

The transmission order information may be set according to the priority in the rendering process of the information processing device 3, for example. For example, the transmission order information may be set so that information on items used first in rendering processing is transmitted first. For example, in rendering processing, an image of an object OB without texture is calculated while changing the viewpoint, and after the viewpoint is determined, an image of an object OB with texture is calculated from this viewpoint. . For example, the information processing device 3 can calculate an image of the textureless object OB viewed from different viewpoints using the shape information and not the texture information. The transmission item information is set, for example, to information specifying that the shape information is transmitted first and the texture information is transmitted after the shape information. For example, when the imaging device 2 transmits the information of each item of the model information in the order according to the priority in the rendering processing of the information processing device 3, the information is sent in parallel with part of the rendering processing of the information processing device 3. can be sent. Further, for example, the control unit 28 of the imaging device 2 can cause at least a part of model information calculation by the information calculation unit 12 and information transmission processing by the communication unit 13 to be executed in parallel.

The control unit 33 of the information processing device 3 causes the storage unit 31 to store information input to the input device 4, for example. Information input to the input device 4 includes, for example, setting information for rendering processing. This setting information includes, for example, target data for drawing processing (e.g., shape information of model information), viewpoint information in drawing processing, object data to which textures are pasted in texture mapping processing, and texture pasting in texture mapping processing. It includes at least one of information (eg, texture information in model information) and light source information in shading processing (eg, light source information in model information). The rendering processing unit 32 executes rendering processing according to, for example, setting information.

The control unit 33 causes the display device 5 to display, for example, an image showing various information stored in the storage unit 31 . For example, the control unit 33 causes the display device 5 to display setting information for rendering processing, and accepts changes to the setting information through the input device 4 . Further, the control unit 33 causes the display device 5 to display an image indicated by data of the estimated image stored in the storage unit 31 .

Note that the information processing device 3 may not cause the display device 5 to display the estimated image obtained by the rendering process, and in this case, the imaging system 1 may not include the display device 5 . For example, the information processing device 3 transmits at least part of the data of the estimated image calculated by the rendering process to another device (playback device) via the communication unit 30, and the other device displays this image. good too. For example, the information processing device 3 may transmit the data of the estimated image to the communication unit 13 of the imaging device 2 via the communication unit 30, and the imaging device 2 transmits the estimated image data received via the communication unit 13. An estimated image may be displayed on the display unit 25 based on the data. For example, the playback device acquires information (estimated image) calculated by rendering processing and displays the information on the display unit.

Note that the information processing device 3 may receive various setting information from another device via the communication unit 30 , and in this case, the imaging system 1 may not include the input device 4 . For example, the imaging device 2 may transmit setting information for rendering processing (for example, viewpoint information from which an estimated image is based) to the communication unit 30 of the information processing device 3 via the communication unit 13 . Further, the information processing device 3 may perform the rendering process according to the setting information for the rendering process received from the imaging device 2 .

Note that the imaging device 2 may transmit a command requesting data of an estimated image calculated by rendering processing to the communication unit 30 of the information processing device 3 via the communication unit 13 . The information processing device 3 may transmit the estimated image data to the communication unit 13 of the imaging device 2 via the communication unit 30 as a response to the command from the imaging device 2 . The imaging device 2 may transmit the request command as a part of the setting information of the rendering process, or may transmit the command (eg, control signal) separate from the setting information of the rendering process.

When the imaging device 2 is controlled by the information processing device 3 to execute the various processes described above, the imaging device 2 does not need to include at least part of the control unit 28, the storage unit 27, the display unit 25, and the input unit 26. good. Further, the imaging device 2 may execute the above-described various processes by user's operation, and may not be controlled by the information processing device 3, for example. For example, the imaging device 2 may execute the various processes described above without receiving a command (control signal) from an external device, and may execute the various processes described above according to a user's operation or a predetermined processing schedule. may The imaging device 2 may transmit the calculation result (eg, model information, etc.) of the information calculation unit 12 to another device of the information processing device 3 via the communication unit 13 .

Next, an image processing method will be described based on the operation of the imaging system 1. FIG. FIG. 4 is a flow chart showing the operation of the imaging system 1. As shown in FIG. In step S1 of FIG. 4, the imaging device 2 captures an image of the object OB using the imaging unit 15, and acquires data of a captured image (for example, a visible light image) (imaging processing). Also, in step S2, the imaging device 2 acquires point cloud data representing the three-dimensional shape of the surface of the object OB (point cloud data processing). In step S2, the imaging device 2 detects the distance between the object OB and the imaging device 2 by the distance measuring unit 16, and acquires depth information (step S3). Further, the information calculation unit 12 of the imaging device 2 calculates the depth information as point cloud data by, for example, perspective transformation (step S4). Further, in step S4, for example, the information calculation unit 12 adjusts the parallax between the data of the visible light image and the depth information as necessary (viewpoint conversion processing). The information calculation unit 12 may perform the processing of step S2 in parallel with at least a part of the processing of step S1, or may perform the processing of step S1 before or after the processing of step S1.

In step S5, the information calculation unit 12 calculates (generates) model information (model information generation processing). The information calculator 12, for example, calculates surface information as shape information (step S6) (surface processing). Further, the information calculation unit 12 calculates texture information by, for example, an inverse rendering method using the calculated surface information (step S7). In step S7, the information calculator 12 calculates light source information (step S8). Further, the information calculation unit 12 calculates pattern information using, for example, the calculated light source information (step S9). In step S5, the information calculation unit 12 causes the storage unit 27 to store the calculated information for each item.

In step S<b>10 , the imaging device 2 transmits at least part of the model information stored in the storage section 27 to the information processing device 3 via the communication section 13 . The information processing device 3 executes rendering processing using, for example, the model information received from the imaging device 2 and causes the display device 5 to display the result (eg, estimated image).

[Second embodiment]
Next, a second embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 5 is a diagram showing the imaging system 1 according to this embodiment. The imaging device 2 according to this embodiment further includes a model integration unit 35 and a position detection unit 36 .

The model integration unit 35 of the imaging device 2 detects first model information calculated based on the result of detecting the object OB from the first direction (first detection result), and detects the object OB from the second direction. Integrated model information is generated by integrating the second model information calculated based on the result (second detection result). In this embodiment, the information calculation unit 12 calculates the first model information and the second model information using the techniques described above.

For example, the model integration unit 35 associates the first feature point of the shape indicated by the first model information with the second feature point of the shape indicated by the second model information, and 2 Integrate model information. A first feature point is a portion of the shape indicated by the first model information that can be distinguished from other portions. For example, a portion defined as a surface in surface information can be distinguished from other surfaces by the shape of its outer circumference. A second feature point is a portion of the shape indicated by the second model information that can be distinguished from other portions.

For example, the model integration unit 35 detects the first feature point using at least one of the shape information and the texture information included in the first model information, and the position of the first feature point and the feature amount of the feature point ( First feature point data indicating the first feature amount) is calculated. The model integration unit 35 stores, for example, the first feature point data in the storage unit 27 as information of one item of the first model information. In addition, the model integration unit 35 detects the second feature points using at least one of the shape information and the texture information included in the second model information, for example, and calculates the positions of the second feature points and the features of the feature points. Second feature point data indicating the amount (second feature amount) is calculated. The model integration unit 35 stores, for example, the second feature point data in the storage unit 27 as information of one item of the second model information. The model integration unit 35 calculates feature points and feature amounts by, for example, a SIFT (Scale-Invariant Feature Transform) method or a SURF (Speeded Up Robust Features) method. Note that, for example, the feature point data (eg, first feature point data, second feature point data) is data that includes at least the feature amount of the feature point, and the feature point and the feature amount The data may include only quantity.

The model integration unit 35 matches the first feature point data having the first feature amount of the first model information and the second feature point data having the second feature amount of the second model information, and integrates the third feature points. Compute integrated model information that represents the data. The third feature point data is, for example, data obtained by matching first feature point data (eg, first feature amount) and second feature point data (eg, second feature amount). The third feature point data includes, for example, a portion of the object OB where the first feature point data and the second feature point data overlap, and models a wider range than either the first partial shape or the second partial shape. It is integrated data that is ringed.

For example, the imaging device 2 detects the object OB from one side (one side) of an arbitrary first direction, calculates the first model information, detects the object OB from the opposite side of the first direction, and calculates the first model information. When the two-model information is calculated, the model integrating section 35 can calculate the integrated model information covering almost the entire circumference of the object OB. Note that the shape indicated by the integrated model information may be a partial model indicating a part of the object OB.

Here, the integrated model information includes, for example, at least one of shape information and texture information. The shape information of the integrated model information is, for example, information obtained by integrating the shape information of the first model information and the shape information of the second model information. Also, the texture information of the integrated model information is, for example, information obtained by integrating the texture information of the first model information and the texture information of the second model information.

Further, the position detection unit 36 detects position information of the imaging device 2 (for example, the detection unit 11) according to a command (detection request signal) from the control unit 28, for example. The position detection unit 36 includes, for example, at least one of an orientation sensor, a GPS sensor, and an acceleration sensor. The position detection section 36 is provided inside the body section 14 , for example, but may be externally attached to the body section 14 . The position detection section 36 may be a unit removable from the main body section 14 .

The position information of the detection unit 11 includes, for example, at least one of the detection direction of the detection unit 11 and global positioning information. The detection direction of the detection unit 11 is, for example, the direction of the optical axis of the imaging optical system 17 of the imaging unit 15 shown in FIG. 3(A). The detection direction of the detection unit 11 is predetermined for, for example, the orientation of the imaging device 2 , and the position detection unit 36 detects the orientation of the imaging device 2 with the orientation sensor, thereby determining the detection direction of the detection unit 11 . to detect Further, the detection unit 11 detects global positioning information using, for example, a GPS sensor. Also, the position information of the detection unit 11 includes the amount of change in the position of the detection unit 11 . The position detection unit 36 can detect the speed of the detection unit 11 by time-integrating the detection result of the acceleration sensor, for example. Further, the position detection unit 36 detects the amount of change in the position of the detection unit 11 by, for example, time-integrating the speed of the detection unit 11 . For example, the detection unit 11 may detect first-accuracy position information using a GPS sensor, and acquire second-accuracy position information, which is higher than the first accuracy, from the detection result of the acceleration sensor.

For example, when causing the detection unit 11 to detect the object OB, the control unit 28 causes the position detection unit 36 to detect the position information of the detection unit 11 . For example, the control unit 28 causes the storage unit 27 to store the detection result of the detection unit 11 in association with the detection result of the position detection unit 36 (position information of the detection unit 11). The information calculation unit 12 calculates model information based on the detection result of the target object OB by the detection unit 11, associates the information of each item of this model information with the position information of the detection unit 11, and stores it in the storage unit 27. . The model information may include position information of the detection unit 11, for example.

The position information of the detection unit 11 is used, for example, to integrate the first model information and the second model information. For example, the model integration unit 35 stores the first position information of the detection unit 11 when the detection unit 11 detects the original data of the first model information, and the first position information of the detection unit 11 when the detection unit 11 detects the original data of the second model information. The first model information and the second model information may be matched and integrated using the second position information of the detection unit 11 . For example, the model integration unit 35 uses the first position information and the second position information to calculate the positional relationship between the first partial shape indicated by the first model information and the second partial shape indicated by the second model information. good too. When the above position information is used to integrate model information, the imaging device 2 or information processing device 3 that performs integration can reduce the amount of calculation required for integration, and can also reduce mismatching in integration. is.

Note that the model integration unit 35 may integrate the first model information and the second model information using the position information of the detection unit 11 and the feature point data including the feature amount. Alternatively, the model integration unit 35 may integrate the first model information and the second model information using the position information of the detection unit 11 without using the feature point data. For example, the model integration unit 35 may integrate the first model information and the second model information using feature point data (eg, feature amount) without using the position information of the detection unit 11 .

By the way, the area adjacent to the edge of the shape indicated by the shape information of the first model information (hereinafter referred to as the out-of-field area) includes, for example, an area that cannot be detected from the first direction. The out-of-view area is, for example, a portion that is not closed in the shape indicated by the first model information. The information calculation unit 12 or the model integration unit 35 may calculate the detection direction of the detection unit 11 capable of acquiring the shape information of the out-of-field area. The imaging device 2 may display this detection direction on the display unit 25 as a candidate for the second direction, for example, after the detection processing of the object OB from the first direction is completed.

Next, an image processing method will be described based on the operation of the imaging system 1. FIG. FIG. 6 is a flow chart showing the operation of the imaging system 1. As shown in FIG. The imaging device 2 performs the process of step S11 of acquiring at least the data of the captured image in a state where the detection direction of the detection unit 11 is set to the first direction. In step S11, the imaging device 2 detects the target object OB from the first direction by the detection unit 11, and acquires visible light image data and point cloud data, for example. In step S12, the information calculation unit 12 calculates the first model information using the detection result of the detection unit 11 in step S11. In step S<b>13 , the information calculation unit 12 causes the storage unit 27 to store the first model information.

After the detection processing by the detection unit 11 is completed in step S11, the imaging device 2 is moved, for example, manually by the user or by a moving device (not shown), and the detection direction of the detection unit 11 is set to a second direction different from the first direction. be done. In step S14, the imaging device 2 detects the object OB from the second direction by the detection unit 11, and acquires visible light image data and point cloud data, for example. In step S15, the information calculation unit 12 calculates the second model information using the detection result of the detection unit 11 in step S14. In step S<b>16 , the information calculation unit 12 causes the storage unit 27 to store the second model information.

In step S17, the model integration unit 35 integrates the first model information and the second model information to calculate integrated model information. In step S17, the model integration unit 35 reads out at least one of the shape information and the texture information of the first model information from the storage unit 27, and uses at least one of the shape information and the texture information to detect the first feature points. (Step S18). Then, the model integration unit 35 calculates first feature point data including at least the feature amount of the first feature point (step S19), and stores the first feature point data in the storage unit 27 (step S20). The model integration unit 35 reads out at least one of the shape information and the texture information of the second model information from the storage unit 27, and detects the second feature points using at least one of the shape information and the texture information (step S21). . Then, the model integration unit 35 calculates second feature point data including at least the feature amount of the second feature point (step S22), and stores the second feature point data in the storage unit 27 (step S23). The model integration unit 35 reads out the first feature point data and the second feature point data from the storage unit 27, and matches these feature point data (step S24). The model integration unit 35 calculates the integrated model information through the matching process in step S24, and stores the integrated model information for each item in the storage unit 27, for example (step S25). Then, in step S<b>26 , the imaging device 2 transmits at least part of the integrated model information stored in the storage section 27 to the information processing device 3 through the communication section 13 . The information processing device 3 executes rendering processing using, for example, the integrated model information received from the imaging device 2 and causes the display device 5 to display the result (eg, an estimated image, etc.).

[Third embodiment]
Next, a third embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 7 is a diagram showing an imaging system 1 according to this embodiment. This imaging system 1 includes a plurality of imaging devices 2 . In the following description, among the plurality of imaging devices 2, one imaging device 2 is referred to as a first imaging device 2a, and the other imaging device 2 is referred to as a second imaging device 2b. Therefore, this imaging system 1 is configured to include at least two imaging devices or a plurality of imaging devices.

The first imaging device 2a detects the object OB from the first direction and calculates the first model information based on the detection result. The first imaging device 2 a supplies the first model information to the information processing device 3 . The second imaging device 2b detects an object from a second direction and calculates second model information based on the detection result. The second imaging device 2 b supplies the second model information to the information processing device 3 . The information processing device 3 integrates the first model information and the second model information, and creates an integrated model based on detection results detected from a plurality of directions (eg, a first direction and a second direction different from the first direction). Calculate information.

Next, each part of the imaging system 1 will be described. FIG. 8 is a block diagram showing the imaging system 1. As shown in FIG. The first imaging device 2a includes a timer 41 (hereinafter referred to as the timer 41a) that detects information (hereinafter referred to as timing information) at which the detection unit 11 (hereinafter referred to as the detector 11a) detects the object OB. ). The timing information includes, for example, information indicating a period (eg, date and time) during which the detection unit 11a performed the detection process. The timing information may include at least one of the start time and the end time of the period during which the detecting unit 11a has performed the detection process.

The timing information is used, for example, when performing model integration using a plurality of detection results with different timings when the object OB is detected by the imaging device. For example, a plurality of pieces of model information may be calculated based on at least two detection results with relatively close detection timings of the object OB among the plurality of detection results, and the calculated pieces of model information may be integrated.

For example, the detection unit 11a stores the detection result in the storage unit 27 (hereinafter referred to as storage unit 27a) in association with the timing information. The information calculation unit 12 (hereinafter referred to as the information calculation unit 12a) calculates the first model information based on the detection result of the detection unit 11a, for example, and stores the calculation result in the storage unit 27a in association with the timing information. .

Also, the second imaging device 2b has, for example, the same configuration as the first imaging device 2a. The detection unit 11 (hereinafter referred to as the detection unit 11b) of the second imaging device 2b detects, for example, the object OB from the second direction, and the detection result is detected by the timer 41 (hereinafter referred to as the timer 41b). It is stored in the storage unit 27 (hereinafter referred to as storage unit 27b) in association with the detected timing information. The information calculation unit 12 (hereinafter referred to as the information calculation unit 12b) of the second imaging device 2b calculates the second model information based on the detection result of the detection unit 11b, for example, and associates the calculation result with the timing information. It is stored in the storage unit 27b.

Note that the timing information may include information on the period during which the detection unit 11 is scheduled to detect the object OB. The first imaging device 2a and the second imaging device 2b may transmit and receive timing information and synchronize with each other using the control unit 28 and the communication unit 13 to detect the object OB.

In this embodiment, the information calculation unit 12a of the first imaging device 2a calculates the first additional information used for integrating the first model information and the second model information. The first additional information is, for example, information obtained by matching feature point data (eg, feature amount), position information of the detection unit 11a, timing information of the imaging device 2, position information of the detection unit 11b, and the second imaging device. 2b includes at least one of the timing information. For example, the first imaging device 2a receives the second model information from the second imaging device 2b, and the information calculation unit 12a of the first imaging device 2a calculates the first feature point data and the The first additional information is calculated using the received second feature point data.

In this embodiment, the information processing device 3 further includes a model integration unit 45 . The model integration unit 45 integrates the first model information from the first imaging device 2a and the second model information from the second imaging device 2b to generate the integrated model information. The algorithm used by the model integration unit 45 to integrate model information may be, for example, the same as that of the model integration unit 35 of the imaging device 2 described in the second embodiment. The communication unit 30 of the information processing device 3 receives the first additional information from the communication unit 13a of the first imaging device 2a, for example, before model integration is performed. The model integration unit 45 of the information processing device 2 performs model integration using, for example, the first model information, the second model information, and the first additional information.

FIG. 9 is a diagram showing the operation of the imaging system 1 according to this embodiment. The information calculation unit 12a of the first imaging device 2a calculates first model information in step S31, and calculates first feature point data including at least the first feature amount in step S32. The information calculation unit 12b of the second imaging device 2b calculates second model information in step S33, and calculates second feature point data including at least the second feature amount in step S34. In step S35, the communication unit 13a of the first imaging device 2a transmits a command (request command) requesting transmission of the second feature point data to the communication unit 13b of the second imaging device 2b. The communication unit 13b of the second imaging device 2b receives the request command from the communication unit 13a of the first imaging device 2a in step S36. The communication unit 13b transmits the second feature point data to the communication unit 13a in step S37. The communication unit 13a receives the second feature point data from the communication unit 13b in step S38. In this way, communication (data communication) of at least feature point data (eg, first feature point data and/or second feature point data) between imaging devices (eg, between a first imaging device and a second imaging device) ), the information processing apparatus 3 can reduce the integration load by calculating information necessary for integration of the model information by each imaging apparatus. Note that the information processing device 3 can also create the feature point data (the first feature point data, the second feature point data, etc.). In this case, for example, the information processing device 3 generates the first feature point data and the second feature point data using each model information received from each imaging device or one of the imaging devices (eg, the first imaging device 2a). do. Also, the above feature point data (eg, first feature point data, second feature point data) can be generated by the first imaging device 2a (or the second imaging device 2b). In this case, for example, the first imaging device 2a generates second feature point data based on the second model information received from the second imaging device 2b.

In step S39, the information calculation unit 12a of the first imaging device 2a matches the first feature point data and the second feature point data (matching processing of the feature portion). Calculate additional information. The first additional information includes, for example, information indicating the positional relationship between the shape indicated by the first model information and the shape indicated by the second model information. The first additional information includes, for example, coordinates of feature points common to the first feature data and the second feature data and/or feature amounts common to the first feature data and the second feature data. The information calculation unit 12a stores the calculated first additional information in the storage unit 27a, for example. The first additional information may be, for example, one item of information in the first model information.

The communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3 in step S41. The communication unit 30 of the information processing device 3 receives the first additional information from the communication unit 13a in step S42.
In step S43, the information processing device 3 causes the first model information from the first imaging device 2a to be in charge of the first model information from the first imaging device 2a and the second model information from the second imaging device 2b to be in charge of the integrated model information. Determine and set a second area of responsibility. For example, the model integration unit 45 selects each assigned area so that the first assigned area and the second assigned area do not overlap.

In step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 to issue a command (first request signal) requesting transmission of the first model information in the first coverage area to the first imaging device 2a. It transmits to the communication part 13a. Hereinafter, the first model information in the first assigned area will be referred to as first designation information. The first designation information is, for example, information about a shape of a part of the shape indicated by the first model information. The communication unit 13a of the first imaging device 2a receives the command of step S44 in step S45. The information calculation unit 12a of the first imaging device 2a extracts the first designation information from the first model information in step S46. The communication unit 13a of the first imaging device 2a transmits the first designation information to the communication unit 30 of the information processing device 3 in step S47. In step S48, the communication section 30 receives the first designation information from the communication section 13a.

In step S49 after the process of step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 to issue a command (second request signal) requesting transmission of the second model information in the second area in charge. , to the communication unit 13b of the second imaging device 2b. Hereinafter, the second model information in the second assigned area will be referred to as second designation information. The second designation information is, for example, information related to the shape of a part of the shape indicated by the second model information. The communication unit 13b of the second imaging device 2b receives the command of step S49 in step S50. The information calculation unit 12b of the second imaging device 2b extracts the second designation information from the second model information in step S51. The communication unit 13b of the second imaging device 2b transmits the second designation information to the communication unit 30 of the information processing device 3 in step S52. The communication unit 30 of the information processing device 3 receives the second designation information from the communication unit 13b in step S53. In step S54, the model integration unit 45 of the information processing device 3 integrates the first designation information, which is part of the first model information, and the second designation information, which is part of the second model information, into an integrated model. Compute information. For example, the model integration unit 45 shares model information (eg, first model information, second model information) or feature point data (eg, first feature point data, second feature point data) among a plurality of imaging devices. Integrated model information is generated using information (eg, additional information, specified information) obtained by (eg, information sharing through two-way communication or one-way communication). Further, for example, the imaging devices 2a and 2b exchange information (eg, first model information, second model information, first feature point data, second feature point data).

[Fourth embodiment]
Next, a fourth embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. The imaging system 1 according to this embodiment has the same configuration as in FIGS. 7 and 8. FIG. In this embodiment, the first imaging device 2a and the second imaging device 2b each calculate additional information used for model integration. In this embodiment, the information processing device 3 checks the integrated model information and appropriately performs interpolation processing.

FIG. 10 is a diagram showing the operation of the imaging system 1 according to this embodiment. In FIG. 10, the same reference numerals are given to the same processes as in FIG. 9, and the description thereof will be simplified or omitted. The information calculation unit 12a of the first imaging device 2a calculates first model information in step S31, and calculates first feature point data including at least the first feature amount in step S32. The information calculation unit 12b of the second imaging device 2b calculates second model information in step S33, and calculates second feature point data including at least the second feature amount in step S34. The communication unit 13a of the imaging device 2 transmits the first feature point data to the communication unit 13b of the second imaging device 2b in step S60. The communication unit 13b receives the first feature point data from the communication unit 13a in step S61. The communication unit 13b transmits the second feature point data to the communication unit 13a in step S62. The communication unit 13a receives the second feature point data from the communication unit 13b in step S63.

In step S64, the information calculation unit 12a of the first imaging device 2a matches the first feature point data and the second feature point data, and calculates first additional information indicating the matching result. The communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3 in step S65. The communication unit 30 receives the first additional information from the communication unit 13a in step S66.

Similar to the first imaging device 2a, the information calculation unit 12b of the second imaging device 2b matches the first feature point data and the second feature point data, and in step S67, second additional information indicating the matching result Calculate The information calculation unit 12b stores the calculated second additional information in the storage unit 27b. The second additional information is the same as the first additional information, and may be information of one item of the second model information, for example. The communication unit 13b of the second imaging device 2b transmits the second additional information to the communication unit 30 of the information processing device 3 in step S68. The communication unit 30 receives the second additional information from the communication unit 13b in step S69.

The imaging system 1 integrates the first model information and the second model information, for example, through the processes of steps S43 to S54 described with reference to FIG. The model integration unit 45 of the information processing device 3 stores the calculated integrated model information in the storage unit 31, for example. In step S70, the model integration unit 45 checks (determines) the consistency of the calculated integrated model information. When the model integration unit 45 determines that the integrated model information is not consistent, the model integration unit 45 performs, for example, the following processing.

In step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 and transmits a command (request signal) requesting transmission of the first interpolation information to the communication unit 13a of the first imaging device 2a. The first interpolation information is, for example, information used for interpolating the integrated model information, and includes information on areas where the shape indicated by the first model information and the shape indicated by the second model information overlap. In step S72, the communication unit 13a of the first imaging device 2a receives the command of step S68. In step S73, the information calculation unit 12a of the first imaging device 2a extracts the first interpolation information from the first model information. In step S<b>74 , the communication section 13 a of the first imaging device 2 a transmits the first interpolation information to the communication section 30 of the information processing device 3 . In step S75, the communication unit 30 receives the first interpolation information from the communication unit 13a.

In step S76 after step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 to issue a command (request signal) requesting transmission of the second interpolation information to the communication unit of the second imaging device 2b. 13b. The second interpolation information is, for example, information used for interpolating the integrated model information, and includes information on areas where the shape indicated by the first model information and the shape indicated by the second model information overlap. In step S77, the communication unit 13b of the second imaging device 2b receives the command of step S76. In step S78, the information calculation unit 12b of the second imaging device 2b extracts the second interpolation information from the second model information. In step S<b>79 , the communication section 13 b of the second imaging device 2 b transmits the second interpolation information to the communication section 30 of the information processing device 3 . In step S80, the communication unit 30 receives the second interpolation information from the communication unit 13b.

In step S81, the model integration unit 45 of the information processing device 3 uses the first interpolation information and the second interpolation information to interpolate the integrated model information. In step S82, the model integration unit 45 updates the integrated model information stored in the storage unit 31 to integrated model information after the interpolation processing.

Also, for example, the model integration unit 45 may combine model information (eg, first model information, second model information) or feature point data (eg, first feature point data, second feature point data) between a plurality of imaging devices. Information (eg, additional information, specified information) obtained by sharing information (eg, sharing information through two-way communication or one-way communication) is used to generate integrated model information.

[Fifth embodiment]
Next, a fifth embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 11 is a block diagram showing the imaging system 1 according to this embodiment.

An imaging system 1 according to the present embodiment includes a first imaging device 2 a, a second imaging device 2 b, and an information processing device 3 . The first imaging device 2a includes, for example, a model integration unit 35 (hereinafter referred to as model integration unit 35a) similar to that of the second embodiment. The model integration unit 35a integrates the first model information calculated by the information calculation unit 12a and the second model information received by the communication unit 13a from the second imaging device 2b. The communication unit 13 a transmits the integrated model information calculated by the model integration unit 35 a to the communication unit 30 of the information processing device 3 .

FIG. 12 is a diagram showing the operation of the imaging system 1 according to this embodiment. In FIG. 12, the same reference numerals are given to the same processing as in FIG. 9, etc., and the description thereof will be simplified or omitted. The imaging system 1 calculates the first additional information used for model integration through the processes of steps S31 to S40.

In step S85, the first imaging device 2a divides the integrated model information into a first coverage area to be assigned to the first model information and a second coverage area to be assigned to the second model information from the second imaging device 2b. Decide and set. For example, the model integration unit 35a selects each assigned area so that the first assigned area and the second assigned area do not overlap.

In step S86, the control unit 28 of the first imaging device 2a controls the communication unit 13a and transmits a command requesting transmission of the second model information in the second coverage area to the communication unit 13b of the second imaging device 2b. do. The communication unit 13b of the second imaging device 2b receives the command of step S86 in step S87. In step S88, the information calculation unit 12b of the second imaging device 2b extracts the information of the second assigned region specified by the first imaging device 2a from the second model information. In step S89, the communication unit 13b of the second imaging device 2b transmits the information (extracted information) extracted by the information calculation unit 12b in step S86 to the communication unit 13a of the first imaging device 2a.

The information calculation unit 12a of the first imaging device 2a extracts the first model information of the first assigned area in step S90. Moreover, the communication part 13a of the 1st imaging device 2a receives extraction information (for example, the 2nd model information of a 2nd area|region in its duty) from the communication part 13b in step S91. In step S92, the model integration unit 35a of the first imaging device 2a integrates the first model information of the first assigned area and the second model information of the second assigned area to calculate integrated model information. The communication unit 13a of the first imaging device 2a transmits the integrated model information to the communication unit 30 of the information processing device 3 in step S93. The communication unit 30 receives the integrated model information from the communication unit 13a in step S94. The rendering processing unit 32 of the information processing device 3 performs rendering processing using the integrated model information in step S95.

Further, for example, the model integration unit 35a of the first imaging device 2a may combine model information (eg, first model information, second model information) or feature point data (eg, first feature point data, Integrated model information is generated using information (eg, additional information, specified information) obtained by sharing (eg, sharing information through two-way communication or one-way communication) the second feature point data).

[Sixth embodiment]
Next, a sixth embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 13 is a block diagram showing the imaging system 1 according to this embodiment.

In this embodiment, the first imaging device 2a and the second imaging device 2b mutually communicate information used for model integration, and respectively calculate integrated model information. The first imaging device 2a calculates integrated model information by the process described with reference to FIG. 12, for example. Also, the second imaging device 2b calculates the integrated model information by, for example, performing the same processing as the first imaging device 2a.

The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and executes rendering processing using the received integrated model information. For example, the communication unit 30 of the information processing device 3 transmits a command requesting transmission of integrated model information to the communication unit 13a of the first imaging device 2a. The control unit 28 of the first imaging device 2 a controls the communication unit 13 a to transmit the information of the item specified in the command among the integrated model information to the communication unit 30 of the information processing device 3 . The information processing device 3 can similarly cause the second imaging device 2b to transmit the integrated model information.

Note that the information processing device 3 monitors, for example, load information (eg, ratio of load to processing power) of each unit of the imaging system 1, and assigns processing related to model integration to a processing unit having a relatively small load in the imaging system 1. may be assigned. For example, the control unit 28 of the first imaging device 2a generates load information of the first imaging device 2a. The load information of the first imaging device 2a includes, for example, the load information of the information calculation unit 12a and the load information of the model integration unit 35a. The communication unit 13 a of the first imaging device 2 a transmits the load information of the first imaging device 2 a to the communication unit 30 of the information processing device 3 . Similarly, the second imaging device 2 b transmits load information of the second imaging device 2 b to the communication unit 30 of the information processing device 3 . The control unit 33 of the information processing device 3 compares the load of each unit based on the received load information, and causes the processing unit with the relatively light load to execute processing (processing related to model integration) on only one imaging device. Alternatively, a plurality of imaging devices are caused to partially share processing (processing related to model integration) and execute it.

For example, when the control unit 33 determines that the load of the model integration unit 35a of the first imaging device 2a is larger than the load of the model integration unit 35b of the second imaging device 2b, the model integration by the first imaging device 2a At least a part may be restricted and the restricted processing may be executed by the second imaging device 2b. For example, the control unit 33 may prohibit the calculation of the first additional information by the first imaging device 2a and cause the second imaging device 2b to transmit the second additional information to the first imaging device 2a. 2a may perform model integration using the second additional information. Further, for example, the control unit 33 may prohibit the determination processing of the assigned area by the first imaging device 2a, and transmit the assigned area information defining the assigned area from the second imaging device 2b to the first imaging device 2a. , the first imaging device 2a may perform model integration using the area-in-charge information. Further, for example, the control unit 33 may prohibit model integration by the first imaging device 2a and receive integrated model information from the second imaging device 2b.

Note that when the control unit 33 determines that the load on the information processing device 3 is smaller than either of the first imaging device 2a and the second imaging device 2b, at least one of the first imaging device 2a and the second imaging device 2b may be restricted at least part of the model integration process of , and the model integration unit 45 may be caused to execute the restricted process. For example, the control unit 33, for example, prohibits the determination processing of the responsible region by each of the first imaging device 2a and the second imaging device 2b, causes the model integration unit 45 to execute the determination processing of the responsible region, and the responsible region information You may make it transmit to each of the 1st imaging device 2a and the 2nd imaging device 2b. Further, the control unit 33 may prohibit the model integration processing by each of the first imaging device 2a and the second imaging device 2b, and cause the model integration unit 45 of the information processing device 3 to execute the model integration processing.

Note that, like the control unit 33 described above, a load control unit (load monitoring unit) that executes at least one of control and monitoring of the load of the imaging system 1 is provided in the first imaging device 2a (eg, the control unit 28). Alternatively, it may be provided in the second imaging device 2b (eg, the control unit 28). Also, the load control section (load monitoring section) may be provided outside the imaging system 1 or may not be provided in the imaging system 1 .

In addition, for example, the model integration unit 35a of the first imaging device 2a (or/and the model integration unit 35b of the second imaging device 2b) may obtain model information (eg, first model information, second model information) among a plurality of imaging devices. information) or information obtained by sharing feature point data (e.g., first feature point data, second feature point data) (e.g., sharing information by two-way communication, one-way communication) information) to generate integrated model information.

[Seventh embodiment]
Next, a seventh embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 14 is a block diagram showing the imaging system 1 according to this embodiment.

An imaging system 1 according to the present embodiment includes a first imaging device 2 a, a second imaging device 2 b, and an information processing device 3 . The first imaging device 2a and the second imaging device 2b are each capable of calculating integrated model information. In this embodiment, the information processing device 3 does not include a model integration unit. The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and executes rendering processing using the received integrated model information.

FIG. 15 is a diagram showing the operation of the imaging system 1 according to this embodiment. In the following description, processing equivalent to that in FIG. 10, etc., will be given the same reference numerals, and the description thereof will be omitted or simplified. The first imaging device 2a calculates first model information (step S31) and calculates first feature point data (step S32). The second imaging device 2b calculates second model information (step S33) and calculates first feature point data (step S34). The communication unit 13a of the first imaging device 2a transmits the first feature point data to the communication unit 13b of the second imaging device 2b (step S60), and the communication unit 13b receives the first feature point data (step S61). ). The communication unit 13b transmits the second feature point data to the communication unit 13a of the first imaging device 2a (step S62), and the communication unit 13a receives the second feature point data (step S63).

The information calculation unit 12a of the first imaging device 2a calculates first additional information using the first feature point data and the second feature point data (step S64). The information calculation unit 12b of the second imaging device 2b calculates second additional information using the first feature point data and the second feature point data (step S67). The communication unit 13a of the first imaging device 2a transmits first additional information to the communication unit 13b of the second imaging device 2b (step S101), and the communication unit 13b receives the first additional information (step S102). The communication unit 13b transmits the second additional information to the communication unit 13a of the first imaging device 2a (step S103), and the communication unit 13a receives the second additional information (step S104).

The model integration unit 35a of the first imaging device 2a determines the assigned area using the first additional information and the second additional information (step S105). The information calculation unit 12a of the first imaging device 2a extracts the first model information of the first assigned area (step S106). The model integration unit 35b of the second imaging device 2b determines the assigned area using the first additional information and the second additional information (step S107). The information calculation unit 12b of the second imaging device 2b extracts the second model information of the second assigned area (step S108).

The communication unit 13a of the first imaging device 2a transmits the first model information of the first coverage area to the communication unit 13b of the second imaging device 2b (step S109), and the communication unit 13b transmits the first model information of the first coverage area. Model information is received (step S110). The communication unit 13b transmits the second model information of the second assigned area to the communication unit 13a of the first imaging device 2a (step S111), and the communication unit 13a receives the second model information of the second assigned area ( step S112).

The model integration unit 35a of the first imaging device 2a performs model integration using the first model information of the first assigned area and the second model information of the second assigned area (step S113). Then, the model integration unit 35a checks the integrated model information (first integrated model information) (step S114). The model integration unit 35b of the second imaging device 2b performs model integration using the first model information of the first assigned area and the second model information of the second assigned area (step S115). Then, the model integration unit 3ba checks the integrated model information (second integrated model information) (step S116).

The model integration unit 35a of the first imaging device 2a and the model integration unit 35b of the second imaging device 2b perform, for example, the following processing when it is determined that the integrated model information is not consistent. The information calculation unit 12a of the first imaging device 2a extracts the first interpolation information (step S117). The information calculation unit 12b of the second imaging device 2b extracts the second interpolation information (step S118).

The communication unit 13a of the first imaging device 2a transmits the first interpolation information to the communication unit 13b of the second imaging device 2b (step S119), and the communication unit 13b receives the first interpolation information (step S120). The communication unit 13b transmits the second interpolation information to the communication unit 13a of the first imaging device 2a (step S121), and the communication unit 13a receives the second interpolation information (step S122).

The model integration unit 35a of the first imaging device 2a performs interpolation processing using the first interpolation information and the second interpolation information (step S123). The model integrating unit 35a updates the integrated model information stored in the storage unit 27a to the integrated model information after interpolation processing (step S124). The model integration unit 35b of the second imaging device 2b performs interpolation processing using the first interpolation information and the second interpolation information (step S125). The model integration unit 35b updates the integrated model information stored in the storage unit 27b to the integrated model information after interpolation processing (step S126).

The information processing device 3 receives information from the first imaging device 2a. For example, the communication unit 30 of the information processing device 3 transmits a command requesting transmission of information used for rendering processing to the communication unit 13a of the first imaging device 2a (step S130). A command is received (step S131). The information calculation unit 12a of the first imaging device 2a extracts the information designated by the command of step S130, for example, from the model information (first model information or integrated model information) (step S134). The communication unit 13a of the first imaging device 2a transmits the information extracted in step S134 to the communication unit 30 of the information processing device 3 (step S135), and the communication unit 30 receives the information transmitted in step S134. (Step S136).

The information processing device 3 receives information from the second imaging device 2b as necessary. For example, when using information held by the second imaging device 2b for rendering processing, the communication unit 30 of the information processing device 3 sends a command requesting transmission of information used for rendering processing to the communication unit of the second imaging device 2b. 13b (step S132), and the communication unit 13b receives the command of step S132 (step S133). The information calculation unit 12b of the second imaging device 2b extracts the information designated by the command of step S132, for example, from the model information (second model information or integrated model information) (step S137). The communication unit 13b of the second imaging device 2b transmits the information extracted in step S137 to the communication unit 30 of the information processing device 3 (step S138), and the communication unit 30 receives the information transmitted in step S138. (Step S139).

The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, the information received from the first imaging device 2a and the information received from the second imaging device 2b. Note that the information processing device 3 does not need to receive information from the second imaging device 2b when the information held by the second imaging device 2b is not used for rendering processing. For example, the communication unit 30 of the information processing device 3 does not have to transmit the command requesting transmission of information to the communication unit 13b of the second imaging device 2b.

Further, for example, the model integration unit 35a of the first imaging device 2a may combine model information (eg, first model information, second model information) or feature point data (eg, first feature point data, Integrated model information (first integrated model information) using information (eg, additional information, specified information) obtained by sharing (eg, sharing information by two-way communication or one-way communication) of second feature point data) to generate For example, the model integration unit 35b of the second image pickup device 2b may extract model information (e.g., first model information, second model information) or feature point data (e.g., first feature point data, second model information) between a plurality of image pickup devices. Generate integrated model information (second integrated model information) using information (eg, additional information, specified information) obtained by sharing feature point data) (eg, sharing information by two-way communication or one-way communication) do. In this way, for example, when each imaging device images the same object OB and generates integrated model information in each imaging device, a plurality of imaging devices (eg, the first imaging device 2a, the second imaging device 2b) may be configured to perform distributed processing of model integration of the entire object OB in each imaging device and generate integrated model information (eg, first integrated model information, second integrated model information) respectively. . In this case, for example, the integrated model information generated by distributed processing in the imaging device is partial integrated model information of the object OB. In such a case, for example, the first integrated model information generated by distributed processing may be a partial integrated model corresponding to 50% of the entire area of the object OB. The integrated model information may share the integrated model information of the entire object OB with each other. Then, in response to a request from the information processing device 3 or a reproduction device such as a device capable of rendering, the imaging device transmits information necessary for rendering (eg, first integrated model information, second integrated model information) to the information processing device. 3 or a playback device.

[Eighth Embodiment]
Next, an eighth embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 16 is a block diagram showing the imaging system 1 according to this embodiment.

In this embodiment, none of the first imaging device 2a, the second imaging device 2b, and the information processing device 3 includes the above-described model integration unit. For example, the information processing device 3 performs the first rendering process using the information (eg, integrated model information) received from the first imaging device 2a, and then performs the first rendering process, and then performs the information (eg, integrated model information) received from the second imaging device 2b. model information) is used to execute the second rendering process. For example, in the first rendering process, the rendering processing unit 32 generates a first estimated image of the object OB viewed from the first viewpoint. For example, in the second rendering process, the rendering processing unit 32 generates a second estimated image of the object OB viewed from the second viewpoint. For example, in generating the second estimated image, the information processing device 3 requests the transmission of information from the second imaging device 2b when the model is missing only with the information held by the first imaging device 2a. . The communication unit 30 of the information processing device 3 may, for example, sequentially receive information used in the second rendering process while the first rendering process is being performed.

In such a configuration, for example, the imaging system 1 in this embodiment can receive information such as model information or additional information so that the information processing device 3 can obtain information necessary for rendering processing of the information processing device 3. and an information processing device (model integration processing device) including a communication unit capable of transmitting information such as the calculated integrated model information to an external device, and a model integration unit that performs model integration processing.

At least one of the first imaging device 2a, the second imaging device 2b, and the information processing device 3 may include a model integration unit. For example, the second imaging device 2b may include the model integration unit, and the first imaging device 2a and the information processing device 3 may not include the model integration unit. In this case, for example, the information processing device 3 may use the information received from the first imaging device 2a to perform the first rendering process described above. Also, the second imaging device 2b may perform at least part of the model integration process while the first rendering process is performed. After the first rendering process, the information processing device 3 may perform the second rendering process using the integrated model information received from the second imaging device 2b.

[Ninth Embodiment]
Next, a ninth embodiment will be described. In this embodiment, the same reference numerals are given to the same configurations as in the above-described embodiment, and the description thereof is simplified or omitted. FIG. 17 is a block diagram showing the imaging system 1 according to this embodiment.

In this embodiment, the communication unit 13 of the imaging device 2 can receive image data of the object OB from an external device (eg, the imaging device 50 as the second imaging device). The imaging device 50 may be a part of the imaging system 1 or may be an external device of the imaging system 1 . The imaging device 50 does not have a distance measuring unit like the imaging device 2, for example. In this case, the imaging device 50 transmits an image having parallax (parallax image) among the images captured by the imaging unit 51 to the imaging device 2 . Then, the imaging device 2 calculates depth information of the imaging device 50 using the received parallax images. The imaging device 50 does not, for example, calculate the model information described above. The imaging device 50 is, for example, a general-purpose camera, a high-performance mobile phone such as a smart phone, a tablet, or the like.

The imaging device 50 includes an imaging unit 51, an input unit 52, a storage unit 53, a display unit 54, a communication unit 55, and a control unit 56, for example. The imaging unit 51 includes, for example, a CCD image sensor and a CMOS image sensor, and is capable of imaging the object OB. The storage unit 53 includes, for example, a non-volatile memory, and stores data of images captured by the imaging unit 51 and various setting information. The input unit 52 receives, for example, commands from the user. The display unit 54 displays, for example, image data stored in the storage unit 53 or images indicated by various setting information. The communication unit 55 can communicate with the communication unit 13 of the imaging device 2 by wire or wirelessly. The communication unit 55 , for example, transmits data of the image of the object OB captured by the imaging unit 51 to the communication unit 13 of the imaging device 2 . The control unit 56 controls each unit of the imaging device 50 .

The information calculation unit 12 of the imaging device 2 calculates the first model information using the detection result of the detection unit 11 as described in the above embodiments. Further, the information calculation unit 12 of the imaging device 2 calculates the second model information based on, for example, the image captured by the imaging unit 15 and/or the image (image data of the object OB) received from the imaging device 50. do. Note that the information calculation unit 12 may not calculate the second model information, and may use, for example, an image received from the imaging device 50 for interpolation processing or the like. For example, the imaging device 2 detects the first depth information by the distance measuring unit 16, calculates the second depth information based on the image captured by the imaging unit 15 and the image received from the imaging device 50, and 2 depth information may be used to interpolate the first depth information.

The communication unit 55 of the imaging device 50 can communicate with the communication unit 30 of the information processing device 3 by wire or wirelessly. The communication unit 55 , for example, transmits setting information for rendering processing (eg, viewpoint information of the estimated image) to the communication unit 30 of the information processing device 3 . The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, model information received from the communication unit 13 of the imaging device 2 and setting information for rendering processing received from the communication unit 55 of the imaging device 30. . The communication unit 30 of the information processing device 30 transmits, for example, data of an estimated image generated by rendering processing to the communication unit 55 of the imaging device 50 . The display unit 54 of the imaging device 50 displays the estimated image based on the estimated image data from the information processing device 30, for example.

Note that the communication unit 55 of the imaging device 50 does not have to communicate with the communication unit 30 of the information processing device 3 . For example, the communication unit 55 may transmit setting information for rendering processing to the communication unit 13 of the imaging device 2 . The communication unit 13 may transmit the setting information received from the communication unit 55 of the imaging device 50 to the communication unit 30 of the information processing device 3 . Further, the communication unit 13 may receive data of an estimated image indicating the result of rendering processing from the communication unit 30 of the information processing device 3 . The communication unit 13 may transmit the estimated image data received from the communication unit 30 of the information processing device 3 to the communication unit 55 of the imaging device 50 . The display unit 54 of the imaging device 50 may display the estimated image based on the estimated image data from the imaging device 2 .

The imaging system 1 or the imaging device 2 of each embodiment described above may be, for example, at least part of a production system or production support system for video content including CG, a visualization imaging system for blood vessels and biological tissues, It may be at least part of a biological microscope system, defect detection system or object metrology system. Also, the imaging system 1 or the imaging device 2 may be at least part of a detection system that detects, for example, the entry of an object, animal, etc. into the viewing area, or the movement of an object, animal, etc. in the viewing area. The detection system may be, for example, a security system, a care system, a surveillance system, a traffic light system, an information acquisition system for acquiring statistical data such as traffic volume, and the like. Also, the imaging system 1 or the imaging device 2 may be at least part of the simulator. This simulator may generate, for example, an estimated image when objects (eg, furniture) and animals (eg, people, pets) are placed in a space (eg, room) modeled by the imaging device 2 . Also, this simulator may generate an estimated image when the object or animal modeled by the imaging device 2 is arranged in a predetermined space (eg, room).

Further, according to each of the above-described embodiments, the imaging system in this embodiment includes the first imaging unit that images the object and the first distance measuring unit that detects the distance from each point on the surface of the object. calculating first model information including at least one of shape information and texture information of an object using a first detection unit and a detection result of the first detection unit, and using the first model information to obtain first feature point data; a first imaging device (2, 2a) comprising a first information calculation unit that generates a second imaging unit that captures an image of an object; and a second measurement that detects the distance from each point on the surface of the object. Second model information including at least one of shape information and texture information of the object is calculated using a second detection unit including the distance portion and the detection result of the second detection unit, and the second model information is used to calculate the second model information. a second image capturing device (2, 2b) comprising a second information calculation unit that generates two feature point data; and first feature point data obtained by data communication between the first image capturing device and the second image capturing device. and a model integration unit (35, 45) that integrates the first model information and the second model information using the second feature point data to generate integrated model information of the object. . In such a configuration, the model integration unit may be provided in at least one of the first imaging device, the second imaging device, or the information processing device, or may be provided in another external terminal (model integration processing device). It's okay to be

It should be noted that the technical scope of the present invention is not limited to the above embodiments or modifications. For example, one or more of the requirements described in the above embodiments or variations may be omitted. Also, the requirements described in the above embodiment or modifications can be combined as appropriate.

Note that the number of imaging devices included in the imaging system 1 may be three or more. For example, the imaging system 1 may include four imaging devices, two of which may include the model integration section and the remaining two may not include the model integration section. For example, an imaging device that includes a model integration unit may receive model information from an imaging device that does not include a model integration unit and perform model integration. In this case, the first integrated model information and the second integrated model information are calculated, and the information processing device 3 may integrate the first integrated model information and the second integrated model information. Note that the imaging device according to the present embodiment may be a wearable terminal, and the main unit includes a mounting unit that can be worn on an arm or head, and an input unit that can perform voice input, input buttons, and touch operations. It's okay to be there.

DESCRIPTION OF SYMBOLS 1... Imaging system, 2... Imaging device, 3... Information processing apparatus, 11... Detection part, 12... Information calculation part, 13... Communication part, 15... Imaging part , 16... ranging unit, 18... image sensor, 19... irradiation unit, 21... image sensor,
35 ... model integration part, OB ... object

Claims (15)

a detection unit that detects an object within a field of view;
an information generation unit that generates shape information of the object based on the detection result of the detection unit;
a communication unit that receives a request signal from an external device and transmits the shape information to the external device;
The communication unit receives the request signal specifying a range narrower than the viewing range,
The information generating unit generates the shape information of the range specified by the request signal,
The detection device, wherein the communication unit transmits the shape information of the range. The communication unit transmits to the external device additional information used for integrating a plurality of pieces of shape information generated by detecting the object from a plurality of positions different from each other,
2. The detection device according to claim 1, wherein said request signal is received after transmitting said additional information. 3. The detection device according to claim 2, wherein said additional information includes information indicating a positional relationship between a plurality of pieces of said shape information generated by detecting said object from a plurality of positions different from each other. 4. The detection device according to claim 2, wherein said additional information includes position information of said detection unit. The detection device according to any one of claims 2 to 4, wherein the additional information includes detection timing information of the detection unit. The detection device according to any one of claims 2 to 5, wherein the additional information is generated using information received from a detection device that detects the object from different positions. The detection device according to any one of claims 1 to 6, wherein said communication unit does not transmit said shape information in a range not specified by said request signal. The detection device according to any one of claims 1 to 7, wherein the detection unit detects distances to a plurality of points on the object. The detection device according to any one of claims 1 to 8, wherein the detection unit detects an image of the object. The detection unit detects distances to a plurality of points on the object and an image of the object, and calculates the distance for each partial area of the image of the object. 10. The detection device according to any one of 9. a first detection device for detecting a captured image of an object and a distance from the object, and generating first shape information indicating a shape of a part, but not all, of a surface of the object; and a captured image of the object. and a second detection device that detects a distance from the object and generates second shape information that indicates a shape of a portion, but not all, of the surface of the object, and an information processing device that communicates,
a receiver that receives additional information indicating a positional relationship between the first shape information and the second shape information;
transmitting a request signal requesting the first shape information within a range determined based on the additional information out of the first shape information to the first detecting device, and determining the range based on the additional information out of the second shape information; a transmission unit that transmits a request signal requesting the second shape information of the range to the second detection device;
Information processing device having 12. The information processing apparatus according to claim 11, wherein said additional information includes timing when said first detection device detects said object and timing when said second detection device detects said object. a first detection device for detecting an object from a first position and generating first shape information of the object;
a second detection device communicable with the first detection device for detecting the object from a second position and generating second shape information for the object;
An information processing device that can communicate with the first detection device and the second detection device and generates integrated shape information that integrates the first shape information and the second shape information,
The information processing device transmits information indicating a portion of the first shape information necessary to generate the integrated shape information to the first detection device, and detects the first shape information necessary to generate the integrated shape information. 2. A system for transmitting information indicative of portions of shape information to said second detection device. 14. The system of claim 13, wherein the first detection device receives information from the second detection device generated when the second detection device detects the object. The first detection device is generated based on information generated when the second detection device detects the object and information generated when the first detection device detects the object. transmitting additional information to the information processing device;
15. The system according to claim 14, wherein said information processing device determines a range of said first shape information necessary to generate said integrated shape information based on said additional information.

Images