JP5471919B2 - Image processing apparatus, image processing program, image processing method, and moving body - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing program, an image processing method, and a moving object.

  In recent years, robots that move autonomously and provide various services have been put into practical use. Among the services provided by robots, there is a service that distributes images around a robot taken by a robot to a remote location. Recently, with the development of communication infrastructure, it has become possible to distribute video shot by a camera wirelessly. Accordingly, in the future, there is a possibility that a service or the like that allows the robot to check images (still images and moving images) taken while moving around the city in real time from the world via the Internet.

  On the other hand, social demands for privacy protection are very high. Therefore, in order to spread the service as described above, it is necessary to limit the portion that can be browsed in the captured image. For example, when a robot moves around a city and shoots, there is a need for a restriction such as making it impossible to browse a personal house shown in the captured image.

  On the other hand, there are techniques as disclosed in Patent Documents 1 and 2 for limiting the browsing of images taken by a camera. The technique of Patent Document 1 is to set a masking setting for a portion of the image captured by a pan / tilt camera having a fixed arrangement position of a surveillance camera or the like using pan / tilt encoder information as a privacy portion. is there. The technique of Patent Document 2 is a technique related to a moving camera. When the position of a mobile phone with a built-in camera is specified using base station data and the mobile phone exists in a certain space, This is to prohibit photography by

JP 2001-69494 A JP 2006-50285 A

  However, in the above-mentioned Patent Document 1, it is assumed that the position of the camera is fixed, and image browsing is limited using only pan / tilt encoder information. That is, the present technology cannot be applied to restrict browsing of images taken by a moving robot.

  Further, in the above-mentioned Patent Document 2, only the setting of permission or prohibition of photographing by the camera can be performed, and it is not possible to prohibit browsing only a part of the area shown on the screen.

  In the technology that restricts browsing, as described above, personal homes are prohibited from browsing, but for objects that do not need to be restricted in front of the house (such as cars) It is desirable to be able to browse. In addition, the position and orientation of the camera mounted on the robot is often estimated using a particle filter. However, even when there are multiple samples (hypotheses) with high likelihood in the estimation, high accuracy is achieved. It is necessary to restrict browsing.

  Accordingly, the present invention has been made in view of the above problems, and provides an image processing apparatus, an image processing program, an image processing method, and a moving body that can perform browsing restriction of captured images with high accuracy. Objective.

  An image processing apparatus described in this specification includes an image acquisition unit that acquires an image from a camera mounted on a moving body, and depth information that detects depth information of the image that is mounted on the moving body. A depth information acquisition unit that acquires the depth information from a detection unit, an acquisition unit that acquires the position and orientation of the camera estimated using a particle filter, and a two-dimensional surface of a viewing-prohibited polyhedron that must be prohibited from browsing A map information management unit that manages map information including internal position information and height information, and in the position and orientation of each sample of the particle filter, the browsing prohibited polyhedron is projected on the image based on the map information, and Depth values at the pixels on which the viewing-prohibited polyhedron is projected are calculated based on the map information, and each pixel calculated at the position and orientation of each sample is calculated. A browsing prohibition depth determination value calculation unit that sets a minimum depth value to be a browsing prohibition depth determination value of each pixel, and the browsing prohibition polyhedron is projected onto the image based on the map information at the position and orientation of each sample. In addition, a mask determination value calculation unit that accumulates a value based on the likelihood of the sample at the time of projection for the pixels onto which the viewing-prohibited polyhedron is projected, and uses the accumulated value as a mask determination value for each pixel. And a browsing image in which browsing of pixels whose browsing prohibition depth determination value is smaller than the depth information and whose mask determination value is larger than a predetermined threshold among the images acquired by the image acquisition unit is prohibited. And a browsing image generation unit that generates the image processing apparatus.

  The image processing program described in this specification detects a depth information of an image acquisition unit that acquires an image from a camera mounted on a moving body, and that constitutes the image mounted on the moving body. A depth information acquisition unit that acquires the depth information from a depth information detection unit, an acquisition unit that acquires a position and orientation of the camera estimated using a particle filter, and a position and orientation of each sample of the particle filter The viewing prohibited polyhedron is projected onto the image based on map information including two-dimensional in-plane position information and height information of the viewing prohibited polyhedron that needs to be prohibited, and the depth at the pixel on which the browsing prohibited polyhedron is projected The value is calculated based on the map information, and the minimum value of the depth value at each pixel calculated at the position and orientation of each sample is calculated. The browsing prohibition depth determination value calculation unit that sets the browsing prohibition depth determination value of each pixel, and the position and orientation of each sample project the browsing prohibition polyhedron onto the image based on the map information, and a sample when the projection is performed A value based on the likelihood is accumulated for the pixels on which the viewing-prohibited polyhedron is projected, and a mask determination value calculation unit that uses the accumulated value as a mask determination value for each pixel, the image acquisition unit acquired Function as a browse image generation unit that generates a browse image that prohibits browsing of pixels in which the browse prohibition depth determination value is smaller than the depth information and the mask determination value is greater than a predetermined threshold. An image processing program to be executed.

  In the image processing method described in this specification, a computer detects an image acquisition step of acquiring an image from a camera mounted on a moving body, and depth information of the image included in the moving body. In the depth information acquisition step of acquiring the depth information from the depth information detection unit, the acquisition step of acquiring the position and orientation of the camera estimated using a particle filter, and the position and orientation of each sample of the particle filter The viewing-prohibited polyhedron is projected onto the image based on map information including two-dimensional in-plane position information and height information of the viewing-prohibited polyhedron that needs to be prohibited. The depth value is calculated based on the map information, and the minimum value of the depth value at each pixel calculated at the position and orientation of each sample The browsing prohibited depth determination value calculating step for setting the browsing prohibited depth determination value of each pixel, and the browsing prohibited polyhedron is projected onto the image based on the map information and projected at the position and orientation of each sample. A value based on the likelihood of the sample is accumulated with respect to the pixels on which the viewing-prohibited polyhedron is projected, and a mask determination value calculation step using the accumulated value as a mask determination value for each pixel; and the image acquisition step Browsing image generation for generating a browsing image in which browsing of pixels whose browsing prohibition depth determination value is smaller than the depth information and whose mask determination value is larger than a predetermined threshold among the acquired images is prohibited And an image processing method for executing the process.

  The moving body described in the present specification is a moving body including a camera that captures an image, a depth information detection unit that detects depth information of the image, and an image processing device described in the present specification. is there.

  The image processing apparatus, the image processing program, the image processing method, and the moving body described in the present specification have an effect that viewing of captured images can be restricted with high accuracy.

1 is a schematic configuration diagram of an image browsing system according to an embodiment. It is a functional block diagram of a mobile robot. FIG. 3A is a diagram illustrating an example of a two-dimensional layout map, and FIG. 3B is a diagram illustrating a data structure of a viewing-prohibited polyhedron. It is a figure which shows the hardware constitutions of an image processing apparatus. It is a flowchart which shows the process of a mobile robot (image processing apparatus). It is a figure for demonstrating the camera position and orientation estimation using a particle filter. It is a figure for demonstrating the likelihood calculation process in the camera position and orientation estimation using a particle filter. It is a figure which shows the specific process of step S30 of FIG. It is a figure which shows the specific process of step S70 of FIG. FIG. 10A to FIG. 10G are diagrams schematically showing the outline of the processing of FIG. It is a figure which shows the specific process of step S80 of FIG. It is FIG. (1) for demonstrating the process of step S40 of FIG. FIG. 6 is a (second) diagram for explaining the process of step S40 of FIG. 5; It is a figure for demonstrating the uncertainty at the time of performing camera position and orientation estimation using a particle filter.

  Hereinafter, an embodiment of an image processing apparatus, an image processing program, an image processing method, and a moving body will be described in detail with reference to FIGS.

  FIG. 1 shows a schematic configuration of the image browsing system 100. As shown in FIG. 1, the image browsing system 100 includes a mobile robot 10 as a moving body and one or more PCs 70 as information processing apparatuses. Mobile robot 10 and PC 70 are connected via a communication line such as the Internet (hereinafter collectively referred to as “Internet”).

  FIG. 2 is a functional block diagram of the mobile robot 10. As shown in FIG. 2, the mobile robot 10 includes a stereo camera 12, a drive unit 14, an encoder 16, and an image processing device 18. The stereo camera 12 has two cameras, and the two cameras capture two images from different viewpoints at the same time. The stereo camera 12 also functions as a depth information detection unit that detects the depth information of each pixel by associating the positions of the objects in the images captured by the two cameras. The image captured by the stereo camera 12 and the detected depth information are output to the image processing device 18. The drive unit 14 includes a wheel, a motor that drives the wheel, a drive control unit that drives and controls the motor, and the like. The encoder 16 is a rotary encoder or the like that detects the amount of rotation of the wheel, and outputs the detected amount of rotation of the wheel to the image processing device 18.

  The image processing apparatus 18 includes an image acquisition unit 20, a depth information acquisition unit 22, a camera position / orientation estimation unit 24 as an acquisition unit, a map information management unit 26, a determination value calculation unit 28, and a browsing image generation unit 30. And a transmission unit 32.

  The image acquisition unit 20 acquires an image captured by one camera of the stereo camera 12. The depth information acquisition unit 22 acquires depth information of each pixel detected by the stereo camera 12. The depth information acquisition unit 22 may acquire two images captured by the stereo camera 12, calculate the depth information of each pixel from the images, and acquire the information. In this case, the depth information acquisition unit 22 also has a function as a depth information detection unit.

  The camera position / orientation estimation unit 24 estimates the position / orientation of the stereo camera 12 based on the depth information acquisition result obtained by the depth information acquisition unit 22, the detection result obtained by the encoder 16, and the map information managed by the map information management unit 26. Do. Note that a particle filter is used to estimate the position and orientation of the stereo camera 12, and details of this method will be described later.

  The map information management unit 26 manages map information in a range in which the mobile robot 10 can move. The map information includes a two-dimensional layout map and two-dimensional in-plane position information and height of a viewing-prohibited polyhedron that includes an object (for example, a private house or the like that needs privacy protection) that needs to be prohibited. Contains information. Specifically, as shown in FIG. 3A, the two-dimensional layout map includes data of a two-dimensional arrangement of a robot travel region (road), a wall, and a browsing prohibited polyhedron. Further, the browsing prohibited polyhedron is managed by the map information management unit 26 in a data structure as shown in FIG. The data in FIG. 3B includes the number of vertices of the viewing-prohibited polyhedron, the X and Y coordinates of each vertex on the layout map, and the height lower limit value and upper limit value of the viewing-prohibited polyhedron. The advantage of adopting such a data structure is that it is only necessary to input a polygon on the two-dimensional layout map used for camera position estimation and attach an attribute of height information to the polygon. The setting is easy.

  Returning to FIG. 2, the determination value calculation unit 28 uses the depth information of each pixel acquired by the depth information acquisition unit 22 and the position and orientation of the sample of the particle filter acquired by the camera position and orientation estimation unit 24. Two determination values are calculated. The two determination values are a browsing prohibition depth determination value and a mask determination value. That is, the determination value calculation unit 28 functions as a browsing prohibition depth determination value calculation unit and a mask determination value calculation unit. Details of these determination values will be described later.

  The browsing image generation unit 30 generates a browsing image using the image acquired by the image acquisition unit 20 and the two determination values calculated by the determination value calculation unit 28. Specifically, the browsing image generation unit 30 generates a browsing prohibition mask value that determines whether browsing is prohibited based on two determination values for each pixel, and uses the browsing prohibition mask value to generate an image. An image (browsing image) in which browsing is prohibited is generated. The transmission unit 32 transmits the browsing image generated by the browsing image generation unit 30 to the PC 70 via the Internet.

  The image processing device 18 is realized by a hardware configuration as shown in FIG. That is, the image processing apparatus 18 includes a CPU 90, a ROM 92, a RAM 94, a storage unit (here, an HDD (Hard Disk Drive)) 96, an input / output unit 97, and the like. It is connected. The image processing device 18 realizes the functions of the respective units shown in FIG. 2 when the CPU 90 executes a program (image processing program) stored in the ROM 92 or the HDD 96. Further, the stereo camera 12 and the encoder 16 shown in FIG. 2 are connected to the input / output unit 97.

  Next, the operation and processing of the mobile robot 10 in the present embodiment will be described in detail based on FIGS. FIG. 5 is a flowchart showing processing of the mobile robot 10 (image processing device 18). As shown in FIG. 5, first, in step S <b> 10, the image acquisition unit 20 acquires an image (brightness value I (h, v) of each pixel) from the stereo camera 12. Further, the depth information acquisition unit 22 acquires depth information Z (h, v) of each pixel from the image acquired from the stereo camera 12. The coordinates (h, v) are coordinate values in the horizontal and vertical directions within the image plane (in the image) of each pixel. It should be noted that the depth information Z (h, v) is + ∞ for pixels for which the stereo camera 12 cannot be associated with the two cameras and depth information cannot be obtained.

  Next, in step S20, the camera position / orientation estimation unit 24 estimates the camera position / orientation using the depth information Z (h, v) of each pixel and the two-dimensional layout map. Specifically, the camera position / orientation estimation unit 24 performs processing as shown in FIG. Here, the camera position / orientation estimation unit 24 uses a particle filter that is a time-series filter for estimating the camera position / orientation. This method is a method for statistically estimating the state of the estimation target (camera), and has an advantage that the state can be estimated robustly with few restrictions on the system model and the observation model.

  In the particle filter, as shown in FIG. 6, first, a state to be estimated is discretized according to a certain probability distribution to generate a certain number of samples (initialization step). Next, the state of each sample is transitioned based on the state transition model (detection result of the encoder 16) (transition process). Here, when the noise is w and the system model is f, the sample state parameter s can be calculated from the following equation (1).

Next, observation is performed using the depth information of each pixel acquired by the depth information acquisition unit 22 (observation step). Here, the observation amount at time t and y _t. Then evaluated using the following equation (2) the likelihood L of each sample using the observed quantity y _t (likelihood calculation step).

  Then, the weight is calculated from the likelihood of each sample, and the weighted average of the state of each sample is taken to estimate the state of the target (camera) (estimation step). Specifically, the weight π is calculated based on the following equation (3), and the estimated amount ε is calculated based on the following equation (4).

  Thereafter, samples are generated in descending order of likelihood and resampling is performed (resampling step). It is possible to estimate the target state by repeatedly performing such processing at each time.

  In this embodiment, in the camera position / orientation estimation, a measurement map (see FIG. 7) obtained by projecting the depth information Z (h, v) of each pixel acquired by the depth information acquisition unit 22 onto the floor surface, 2 Check against the 3D layout map. Then, the position and orientation of the camera are estimated by a particle filter using the degree of coincidence of both maps as the likelihood. Note that if the matching of the measurement map and the two-dimensional layout map does not match and the likelihood of all the samples is lower than a predetermined threshold value, it is handled as a state in which the position is lost (lost state).

  Returning to FIG. 5, in the next step S30, a subroutine for generating a browsing prohibition mask value M (h, v) for each pixel is executed. In step S30, specifically, the process of FIG. 8 is executed.

  In the process of FIG. 8, first, in step S60, the determination value calculation unit 28 determines whether or not the position has been lost. If the determination here is negative, the process proceeds to step S70, where the determination value calculation unit 28 determines a browsing prohibition depth determination value Z_Inh (h, v) and a mask determination value E (h, v). (FIG. 9) is executed. The viewing prohibition depth determination value Z_Inh (h, v) is a value indicating the depth of the viewing prohibition polyhedron projected onto the pixel at the coordinates (h, v) on the image plane (on the image). The mask determination value is a determination value indicating the likelihood that the browsing prohibited polyhedron is projected on the pixel.

  In the process of FIG. 9, first, in step S702, the determination value calculation unit 28 initializes the mask determination value E (h, v) and the browsing prohibition depth determination value Z_Inh (h, v). Here, the determination value calculation unit 28 is initialized by setting the mask determination value E (h, v) to 0, and is initialized by setting the browsing prohibition depth determination value Z_Inh (h, v) to + ∞. To do.

  Next, in step S704, the determination value calculation unit 28 selects an unselected particle filter sample S.

Next, in step S706, the determination value calculation unit 28 determines whether the likelihood L _(S) of the sample S (see the above equation (2)) is larger than a predetermined threshold Lth. When judgment here is denied, it transfers to step S724. In addition, the case where the determination here is denied means that the sample is a sample that does not need to be considered in the determination as to whether or not to prohibit browsing because the likelihood is too small.

  On the other hand, if the determination in step S706 is positive, the process proceeds to step S708. In step S708, the determination value calculation unit 28 selects an unselected browsing-prohibited polyhedron H. In this case, the determination value calculation unit 28 extracts one of the browsing prohibited polyhedrons included in the image acquired by the image acquisition unit 20 from the map information managed by the map information management unit 26. select.

  Next, in step S710, the determination value calculation unit 28 selects an unselected surface P of the viewing prohibited polyhedron H selected in step S708.

  Next, in step S712, the determination value calculation unit 28 selects an unselected pixel among the pixels of the image onto which the surface P selected in step S710 is projected.

  Next, in step S714, the determination value calculation unit 28 calculates the depth value Zp (h, v) of the selected pixel. That is, the depth of the selected pixel among the pixels onto which the surface P of the viewing prohibited polyhedron H is projected is calculated based on the data on the surface P (see FIG. 3B).

Next, in step S716, the determination value calculation unit 28 updates the mask determination value E (h, v) and the browsing prohibition depth determination value Z_Inh (h, v). Specifically, the mask determination value E (h, v) is updated based on the following equation (5), and the browsing prohibition depth determination value Z_Inh (h, v) is updated based on the following equation (6).
E (h, v) = E (h, v) + L _(S) (5)
Z_Inh (h, v) = min (Zp (h, v), Z_Inh (h, v)) (6)

In the above equation (5), the mask judgment value E (h, v) is obtained by adding the likelihood L _(S) to the mask judgment value E (h, v) (here, the initial value 0) until immediately before. Update. Further, in the above equation (6), the browsing prohibition depth determination value Z_Inh (h, v) (here, the initial value + ∞) up to immediately before is compared with the depth value Zp (h, v) calculated in step S714. Then, the viewing prohibition depth determination value Z_Inh (h, v) is updated with the smaller one of the values.

  Next, in step S718, the determination value calculation unit 28 determines whether all the pixels of the image on which the surface P is projected have been selected. If the determination is negative, the process returns to step S712, and the process for the next pixel is performed in the same manner as described above. On the other hand, when judgment here is affirmed, it transfers to step S720.

  When the process proceeds to step S720, the determination value calculation unit 28 determines whether or not the entire surface of the viewing prohibited polyhedron H has been selected. If the determination is negative, the process returns to step S710, and the process for the next surface is performed in the same manner as described above. On the other hand, when judgment here is affirmed, it transfers to step S722.

  When the process proceeds to step S722, the determination value calculation unit 28 determines whether all the browsing-prohibited polyhedra H are selected. If the determination here is negative, the process returns to step S708, and the process for the next viewing-prohibited polyhedron is performed in the same manner as described above. On the other hand, when judgment here is affirmed, it transfers to step S724.

  When the process proceeds to step S724 (when the determination at step S722 is affirmed or when the determination at step S706 is negative), the determination value calculation unit 28 determines whether all the samples S have been selected. If the determination here is negative, the process returns to step S704, and processing using the sample S that has not been selected is performed in the same manner as described above. On the other hand, if the determination here is affirmed, the entire processing of FIG. 9 (subroutine of step S70) is terminated. When the process of step S70 is completed as described above, the process proceeds to step S80 of FIG.

Here, the processing of FIG. 9 will be described using a specific example. 10A to 10G schematically show the outline of the processing of FIG. As shown in FIG. 10A, it is assumed that samples S1 and S2 are acquired as samples with high likelihood as a result of camera position and orientation estimation using a particle filter. In this case, first, the surface P1 of the viewing prohibited polyhedron is evaluated in the sample S1. FIG. 10B shows a state in which the surface P1 is projected onto the image surface (a state in which the surface P1 is projected onto the image). Here, it is assumed that the depth value Zp (h, v) of the plane P1 is 2, and the likelihood L _(S1) of the sample S1 is 0.2. Under such circumstances, as shown in FIG. 10C, the mask determination value E (h, v) is updated to 0.2 for the black pixel portion of FIG. 10B and 0 for the other pixels. Is maintained.

  On the other hand, as shown in FIG. 10 (d), the black pixel portion in FIG. 10 (b) is updated to 2 and the other pixels are maintained at + ∞, as the browsing prohibited depth determination value Z_Inh (h, v). The

Next, the surface P1 of the browsing prohibited polyhedron is evaluated in the sample S2. FIG. 10E shows a state where the surface P1 is projected onto the image surface (a state where the surface P1 is projected onto the image). Here, it is assumed that the depth value Zp (h, v) of the plane P1 is 1, and the likelihood L _(S2) of the sample S2 is 0.3. Under such circumstances, the mask judgment value E (h, v) is updated by adding 0.3 to the black pixel portion of FIG. 10E as shown in FIG. The pixel remains 0. That is, among the black pixels shown in FIG. 10 (e), the mask judgment value E (h, v) of the pixels other than the black pixels shown in FIG. 10 (b) becomes 0.3, and the black pixels shown in FIG. 10 (b). The pixel mask determination value E (h, v) is 0.5.

  On the other hand, as shown in FIG. 10 (g), all of the black pixels in FIG. 10 (e) are updated to 1, and the other pixels are maintained at + ∞, as the browsing prohibited depth determination value Z_Inh (h, v). Is done.

  In the process of FIG. 9 of the present embodiment, the process as described above is also performed on the surface P2.

  When the process of FIG. 9 is completed, the process proceeds to step S80 of FIG. In step S80, the determination value calculation unit 28 executes a subroutine (FIG. 11) for determining the browsing prohibition mask value M (h, v).

  In the process of FIG. 11, first, in step S802, the determination value calculation unit 28 selects an unselected pixel. Next, in step S804, the determination value calculation unit 28 determines whether or not the mask determination value E (h, v) is larger than the threshold value Eth. When judgment here is affirmed, it transfers to step S806. Note that the case where the determination here is affirmed means that the pixel selected in step S802 is a pixel on which the same surface of the viewing prohibited polyhedron is projected in one or more samples having a high likelihood. . On the other hand, the case where the determination here is negative means that the pixel selected in step S802 is a pixel in which the same surface of the viewing prohibited polyhedron is not projected in one or more samples having a high likelihood. To do.

  In step S806, the determination value calculation unit 28 determines whether Z_Inh (h, v) is smaller than Z (h, v). When judgment here is affirmed, it transfers to step S808. Note that the case where the determination here is affirmed means that the surface of the viewing-prohibited polyhedron is captured at the pixel selected in step S802. On the other hand, the case where the determination here is negative means that another object existing in front of the viewing-prohibited polyhedron (camera side) is captured at the pixel selected in step S802. .

  In step S808, the determination value calculation unit 28 sets the viewing prohibition mask value M (h, v) to 1.

  On the other hand, if the determinations in steps S804 and S806 are negative, the process proceeds to step S812. In step S812, the determination value calculation unit 28 sets the browsing prohibition mask value M (h, v) to 0.

  Thereafter, after the processing of step S808 or step S812 and the process proceeds to step S810, the determination value calculation unit 28 determines whether or not all the pixels have been selected. If the determination here is negative, the process returns to step S802, and the same processing as described above is executed for the remaining pixels. Then, when the determination in step S810 is affirmed, the entire process in FIG. 11 and the entire process in FIG. 8 are terminated, and the process proceeds to step S40 in FIG.

  By performing the processing of FIG. 11 as described above, the viewing prohibition mask value M (h, v) of the pixel corresponding to the surface of the viewing prohibited polyhedron is set to 1, and the browsing prohibition mask value M of other pixels is set. (H, v) is set to zero. That is, for example, when another object exists in front of the viewing prohibited polyhedron, the viewing prohibition mask value M (h, v) of the pixel corresponding to the other object is set to 0. .

  If the determination in step S60 of FIG. 8 is affirmed, that is, if the position is lost, the process proceeds to step S90. In step S90, the determination value calculation unit 28 sets the viewing prohibition mask value M (h, v) for all pixels to 1, and the process proceeds to step S40 in FIG.

  Returning to FIG. 5, in step S <b> 40, the browsing image generation unit 30 removes the browsing prohibition portion using the image (the luminance value of each pixel) I (h, v) and the browsing prohibition mask value M (h, v). The generated image Im (h, v) is generated. Specifically, as illustrated in FIG. 12, the browsing image generation unit 30 uses an image (viewing prohibition mask) in which a pixel whose browsing prohibition mask value M (h, v) is 1 is a black pixel as an image of the shooting prohibition mask. Image). Then, the browse image generation unit 30 combines the browse prohibition mask image and the image I (h, v) to generate an image Im (h, v) in which the browse prohibition portion is not browseable. That is, here, the browsing image generation unit 30 sets Im (h, v) = I (h, v) when M (h, v) = 0 and M (h, v) = 1. , Im (h, v) = 0.

  Here, for example, as shown in FIG. 13, when an object (a person in FIG. 13) is present in front of the viewing-prohibited polyhedron, the viewing-prohibited mask value M (h, v) of the pixel corresponding to the person is shown. ) Is set to zero. Therefore, in the present embodiment, it is possible to browse a person located in front of the browsing prohibited polyhedron in the browsing image Im (h, v).

  Returning to FIG. 5, in the next step S50, the transmitting unit 32 distributes the image Im (h, v) generated in step S40 to the PC 70 via the Internet. The PC 70 can browse the distributed images (including moving images) almost in real time.

As described above in detail, according to the present embodiment, the image acquisition unit 20 acquires the image I (h, v) from the stereo camera 12 mounted on the mobile robot 10, and the depth information acquisition unit 22 The depth information Z (h, v) constituting the image obtained by the stereo camera 12 is acquired, and the camera position and orientation estimation unit 24 acquires the position and orientation of the camera estimated using the particle filter. . Then, the determination value calculation unit 28 projects the viewing prohibited polyhedron onto the image based on the map information managed by the map information management unit 26 at the position and orientation of each sample S of the particle filter, and the viewing prohibited polyhedron is projected. The depth value Zp (h, v) in the pixel is calculated based on the map information, and the minimum value of the depth value Zp (h, v) in each pixel calculated in the position and orientation of each sample is set as the viewing prohibited depth of each pixel. The determination value is Z_Inh (h, v). In addition, the determination value calculation unit 28 obtains a value based on the likelihood of the sample (in this embodiment, the value of the likelihood L _(S) itself ₎ when the browsing-prohibited polyhedron is projected on the image in the position and orientation of each sample. The viewing-prohibited polyhedron is accumulated for the projected pixels, and the accumulated value is set as a mask determination value E (h, v) for each pixel. Then, the browsing image generation unit 30 is a pixel in which the browsing prohibition depth determination value Z_Inh (h, v) is smaller than the depth information Z (h, v) in the image I (h, v) and the mask determination value E. A browsing image Im (h, v) in which browsing of pixels in which (h, v) is larger than a predetermined threshold Eth is prohibited is generated. Thereby, in this embodiment, a pixel having a high possibility of imaging a viewing-prohibited polyhedron among pixels having a relatively large mask determination value E (h, v) calculated in consideration of the likelihood of the sample. (Pixels whose depth information is greater than or equal to the browsing prohibition depth determination value) can be prohibited from browsing. Thereby, it is possible to generate a browsing image in which privacy or the like is protected.

  In this case, as shown in FIG. 14, there are a plurality of samples (hypotheses) that are likely to be the position and orientation of the camera, and if the weighted average is taken, the estimated position of the camera is an intermediate position between the two samples, and the uncertainty May occur. Even in such a case, by using the mask determination value E (h, v), it is possible to reliably prevent browsing of an object that should be prohibited from being viewed from any position of the plurality of likely positions and orientations. Is possible. Further, in this embodiment, pixels whose depth information is greater than or equal to the browsing prohibition depth determination value are prohibited from browsing, and pixels whose depth information is smaller than the browsing prohibition depth determination value can be browsed. This makes it possible to browse the object (polyhedron) that you want to prohibit browsing, while you can browse the object that is in front of the object (polyhedron) you do not want to prohibit browsing (See FIG. 13).

  In the present embodiment, the image processing apparatus 18 includes a transmission unit 32 that transmits the browsing image Im (h, v) generated by the browsing image generation unit 30 to the PC 70. Thereby, it is possible to make it possible to browse on the PC 70 an image in which a portion that needs to be restricted (for example, a portion that needs to protect privacy) is prohibited.

  In the present embodiment, depth information of an image is acquired from two different images captured by the stereo camera 12. Thereby, a common image can be used as an image acquired by the image acquisition unit 20 and an image for acquiring depth information of each pixel. Therefore, highly accurate depth information of each pixel can be acquired with a simple configuration.

  In the present embodiment, when the camera position / orientation estimation unit 24 cannot estimate the position / orientation of the camera (when the camera is in a lost state), the browsing image generation unit 30 prohibits browsing of all pixels. Generate an image. Thereby, even if the position and orientation of the camera become unknown, it is possible to make it impossible to view at least a part of the image that needs to be restricted (for example, a privacy protection part).

  In the above embodiment, the stereo camera 12 is used as the camera, and the image I (h, v) and the depth information Z (h, v) of each pixel are acquired from the image captured by the stereo camera 12. explained. However, the present invention is not limited to this. For example, a depth information detection unit (Depth Imager or the like) that can detect a depth information Z (h, v) of each pixel while using a camera that can capture one image. ) May be used in combination.

In the above embodiment, the case where the likelihood L _(S) itself is accumulated as shown in the above equation (5) when the mask determination value E (h, v) is updated has been described. It is not something that can be done. For example, as shown in the following equation (7), the square of L _(S) may be accumulated.
E (h, v) = E (h, v) + L _(S) ² (7)

Or it is good also as accumulating the function f (L _(S) ) which makes L _(S) a variable like following Formula (8).
E (h, v) = E (h, v) + f (L _(S) ) (8)

  In the above embodiment, the case where Im (h, v) is set to 0 when the viewing prohibition mask value M (h, v) is 1 has been described, but the present invention is not limited to this. For example, a process such as mosaic or blurring may be performed on a pixel having a browsing prohibition mask value M (h, v) of 1.

  In the above embodiment, the case where the camera position / orientation estimation unit 24 is provided in the image processing apparatus 18 has been described. However, the present invention is not limited to this. For example, a camera position / orientation estimation unit is provided outside the image processing apparatus 18, and a function for acquiring the position and orientation of a stereo camera estimated by the camera position / orientation estimation unit (function as an acquisition unit) is provided in the image processing apparatus 18. ) Only. Even if it does in this way, the effect similar to the said embodiment can be acquired.

  In the above embodiment, the case where the camera is mounted on the mobile robot 10 has been described, but the present invention is not limited to this. The camera may be mounted on a moving body such as an automobile.

  In the above embodiment, the case where the browsing prohibited polyhedron is a building that needs to protect privacy such as a private house has been described. However, the present invention is not limited to this, and an object that should not be made viewable for any reason other than privacy (for defense, confidentiality, legal, educational or ethical reasons) may be set as a viewing-prohibited polyhedron. Good.

  In the above embodiment, the case where the image processing apparatus 18 is mounted on the mobile robot 10 has been described. However, the present invention is not limited to this. For example, the image processing device 18 may communicate with the mobile robot 10 and may not be mounted on the mobile robot 10. In this case, the image processing apparatus 18 may be connected to the Internet of FIG. Further, the image processing device 18 may be incorporated in at least one PC 70 connected to the Internet. That is, a server computer connected to a communication network such as the Internet is used as the image processing apparatus of the present invention, and a service for displaying an image is provided from the server computer to an information processing apparatus such as a personal computer connected thereto. Good (ASP (Application Service Provider)).

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the processing apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium.

  When the program is distributed, for example, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.

10 Mobile robot (moving body)
12 Stereo camera (camera, depth information detector)
DESCRIPTION OF SYMBOLS 18 Image processing apparatus 20 Image acquisition part 22 Depth information acquisition part 24 Camera position and orientation estimation part (acquisition part)
26 Map information management unit 28 Determination value calculation unit (browsing prohibited depth determination value calculation unit, mask determination value calculation unit)
30 browsing image generation unit 32 transmission unit 70 PC (information processing apparatus)

Claims (7)

An image acquisition unit for acquiring an image from a camera mounted on a moving object;
A depth information acquisition unit that acquires the depth information from a depth information detection unit that is mounted on the moving body and detects depth information of each pixel constituting the image;
An acquisition unit that acquires the position and orientation of the camera estimated using a particle filter;
A map information management unit for managing map information including two-dimensional in-plane position information and height information of a browsing prohibited polyhedron that needs to be prohibited from browsing;
At the position and orientation of each sample of the particle filter, the viewing prohibited polyhedron is projected onto the image based on the map information, and a depth value at a pixel on which the viewing prohibited polyhedron is projected is calculated based on the map information. A browsing prohibition depth determination value calculation unit that sets a minimum depth value of each pixel calculated in the position and orientation of each sample as a browsing prohibition depth determination value of each pixel;
At the position and orientation of each sample, the viewing-prohibited polyhedron is projected onto the image based on the map information, and a value based on the likelihood of the sample at the time of projection is projected to the pixel on which the viewing-prohibited polyhedron is projected. A mask determination value calculation unit that accumulates the accumulated value for each pixel and uses the accumulated value as a mask determination value for each pixel;
Of the images acquired by the image acquisition unit, generates a browsing image in which browsing of pixels whose browsing prohibition depth determination value is smaller than the depth information and whose mask determination value is larger than a predetermined threshold is prohibited. A browsing image generation unit to perform,
An image processing apparatus.   The image processing device according to claim 1, further comprising: a transmission unit that transmits the browse image generated by the browse image generation unit to an information processing device. The camera mounted on the moving body is a stereo camera,
The depth information detection unit mounted on the moving body detects depth information of each pixel constituting the image from two different images captured by the stereo camera. An image processing apparatus according to 1. When the acquisition unit cannot acquire the position and orientation of the camera,
The image processing apparatus according to claim 1, wherein the browsing image generation unit generates an image in which browsing of all pixels is prohibited. Computer
An image acquisition unit that acquires images from a camera mounted on a moving object,
A depth information acquisition unit that acquires the depth information from a depth information detection unit that detects depth information of each pixel that constitutes the image, mounted on the moving body,
An acquisition unit for acquiring the position and orientation of the camera estimated using a particle filter;
The browsing prohibited polyhedron is projected onto the image based on map information including two-dimensional in-plane position information and height information of a browsing prohibited polyhedron that needs to be prohibited browsing in the position and orientation of each sample of the particle filter. In addition, the depth value in the pixel on which the browsing prohibited polyhedron is projected is calculated based on the map information, and the minimum depth value in each pixel calculated in the position and orientation of each sample is determined as the browsing prohibited depth determination for each pixel. A browsing prohibition depth judgment value calculation unit,
At the position and orientation of each sample, the viewing-prohibited polyhedron is projected onto the image based on the map information, and a value based on the likelihood of the sample at the time of projection is projected to the pixel on which the viewing-prohibited polyhedron is projected. A mask determination value calculating unit that accumulates the accumulated value and uses the accumulated value as a mask determination value for each pixel;
Of the images acquired by the image acquisition unit, generates a browsing image in which browsing of pixels whose browsing prohibition depth determination value is smaller than the depth information and whose mask determination value is larger than a predetermined threshold is prohibited. Browsing image generation unit,
An image processing program that functions as an image processing program. Computer
An image acquisition step of acquiring an image from a camera mounted on the moving body;
A depth information acquisition step of acquiring the depth information from a depth information detection unit that detects depth information of each pixel constituting the image mounted on the moving body;
An acquisition step of acquiring the position and orientation of the camera estimated using a particle filter;
The browsing prohibited polyhedron is projected onto the image based on map information including two-dimensional in-plane position information and height information of a browsing prohibited polyhedron that needs to be prohibited browsing in the position and orientation of each sample of the particle filter. In addition, the depth value in the pixel on which the browsing prohibited polyhedron is projected is calculated based on the map information, and the minimum depth value in each pixel calculated in the position and orientation of each sample is determined as the browsing prohibited depth determination for each pixel. A browsing prohibition depth determination value calculation step as a value;
At the position and orientation of each sample, the viewing-prohibited polyhedron is projected onto the image based on the map information, and a value based on the likelihood of the sample at the time of projection is projected to the pixel on which the viewing-prohibited polyhedron is projected. A mask determination value calculation step of accumulating and using the accumulated value as a mask determination value for each pixel;
Among the images acquired in the image acquisition step, a browsing image in which browsing of pixels whose browsing prohibition depth determination value is smaller than the depth information and whose mask determination value is larger than a predetermined threshold is prohibited. A browsing image generation process to generate;
An image processing method for executing. A camera for taking images,
A depth information detection unit for detecting depth information of each pixel constituting the image;
A moving body comprising: the image processing apparatus according to claim 1.

Images