JP2019020139A - Information processor, program, and information processing system - Google Patents

Abstract

To increase the convenience of a technique for using imaging information.SOLUTION: An information processor 30 includes a communication unit 31 and a control unit 33. The communication unit 31 acquires imaging information for a predetermined field of vision and distance information for the field of vision. The control unit specifies ones of the pixels in the imaging information on the basis of the distance information, and executes predetermined processing, using the imaging information other than the specified ones.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an information processing device, a program, and an information processing system.

  Conventionally, a technique using imaging information generated by a camera or the like is known. For example, Patent Literature 1 discloses a stereo camera device that calculates a distance to an object using a pair of images captured by two cameras.

JP 2011-191905 A

  Conventionally, it is desired to improve the convenience of a technique using imaging information.

  The objective of this indication is to improve the convenience of the technique using imaging information.

  An information processing apparatus according to an embodiment of the present disclosure includes a communication unit and a control unit. The communication unit acquires imaging information of a predetermined visual field and distance information of the visual field. The control unit identifies a part of the plurality of pixels included in the imaging information based on the distance information, and performs a predetermined process using the imaging information excluding the identified part of the pixel. Run.

  A program according to an embodiment of the present disclosure includes a step of acquiring imaging information of a predetermined visual field and distance information of the visual field in an information processing device, and a part of pixels among the plurality of pixels included in the imaging information. A step of specifying based on the distance information and a step of executing a predetermined process using the imaging information excluding the specified part of the pixels are executed.

  An information processing system according to an embodiment of the present disclosure includes an information generation device and an information processing device. The information generation device generates imaging information of a predetermined visual field and distance information of the visual field. The information processing apparatus identifies a part of a plurality of pixels included in the imaging information based on the distance information, and performs predetermined processing using the imaging information excluding the identified part of the pixels. Execute.

  According to an embodiment of the present disclosure, the convenience of a technique using imaging information is improved.

1 is a block diagram illustrating a schematic configuration of an information processing system according to an embodiment of the present disclosure. It is a figure explaining the visual field of an information generation device. It is a figure explaining the visual field of an information generation device. It is a sequence diagram which shows operation | movement of an information processing system. It is a flowchart which shows operation | movement of the information processing apparatus which performs a foreign material detection process and a foreign material classification process.

  Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

  With reference to FIG. 1, an information processing system 10 according to an embodiment of the present disclosure will be described. The information processing system 10 includes an information generation device 20 and an information processing device 30. The information generation device 20 and the information processing device 30 are connected to be communicable with each other. The information processing system 10 may be mounted on a moving body such as a vehicle, for example.

  The information generation device 20 includes an optical member 21, a first information generation unit 22, a second information generation unit 23, a communication unit 24, a storage unit 25, and a control unit 26.

  The optical member 21 may include, for example, at least one of a housing window that transmits electromagnetic waves, a lens, a filter, a cover, and glass, and a mirror that reflects electromagnetic waves. The optical member 21 is shared by the first information generation unit 22 and the second information generation unit 23. Specifically, electromagnetic waves taken into the information generating device 20 through the optical member 21 are used by the first information generating unit 22 and the second information generating unit 23, respectively.

  The first information generation unit 22 may include a camera, for example. Specifically, the first information generation unit 22 may include an image sensor and a processor.

  In the present disclosure, the “imaging device” may include, for example, a CCD (Charge Coupled Device) imaging device and a CMOS (Complementary Metal Oxide Semiconductor) imaging device. The “processor” may include a general-purpose processor and a dedicated processor specialized for specific processing. The dedicated processor may include a DSP (Digital Signal Processor) and an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA).

  The first information generation unit 22 generates imaging information of the first visual field using light taken into the information generation device 20 via the optical member 21.

  In the present disclosure, the “field of view” may include, for example, a region visible from the viewpoint on an arbitrary virtual plane separated from the viewpoint in the viewing direction. For example, in FIG. 2, a region included in the angle of view of the first information generation unit 22 on the virtual plane P substantially perpendicular or perpendicular to the optical axis 220 of the first information generation unit 22 is the first visual field 221.

  The imaging information of the first visual field 221 may include a plurality of first pixels that respectively correspond to a plurality of regions into which the first visual field 221 is divided. The first pixel may indicate RGB color, for example. Specifically, the pixel value of the first pixel may indicate, for example, RGB color signal intensity. The imaging information may include a captured image, for example. The first information generation unit 22 may periodically generate imaging information of the first visual field 221.

  The second information generation unit 23 illustrated in FIG. 1 may include a laser radar, for example. The laser radar may include a scanning laser radar that two-dimensionally scans a beam-shaped electromagnetic wave, and a flash laser radar that receives a reflected wave or diffused wave of an electromagnetic wave irradiated over a wide range with a distance image sensor. Specifically, the second information generation unit 23 may include a light source, a light receiving element, and a processor.

  In the present disclosure, the “light source” may include any device capable of outputting electromagnetic waves. For example, the light source may include an LED (Light Emitting Diode) light source and a laser light source. The electromagnetic wave may include, for example, infrared light and ultraviolet light. The infrared light may include near infrared light and far infrared light. The “light receiving element” may include any device that can detect electromagnetic waves. For example, the light receiving element may include a photodiode and a distance image sensor.

  The second information generation unit 23 emits the electromagnetic wave output by the light source to the outside of the information generation device 20 via the optical member 21. The second information generation unit 23 generates distance information of the second visual field using the reflected wave or scattered wave of the electromagnetic wave taken into the information generation device 20 via the optical member 21.

  At least a part of the second visual field may overlap with the first visual field 221. For example, in FIG. 2, the region included in the angle of view of the second information generation unit 23 on the virtual plane P is the second visual field 231. A region on the virtual plane P where the first visual field 221 and the second visual field 231 overlap is an overlapping region 40. The surface region 210 of the optical member 21 is included in both the angle of view of the first information generation unit 22 and the angle of view of the second information generation unit 23. The second information generation unit 23 may be disposed in the vicinity of the first information generation unit 22 inside the information generation device 20. The optical axis 230 of the second information generation unit 23 and the optical axis 220 of the first information generation unit 22 may be substantially parallel or parallel.

  The distance information of the second visual field 231 may include a plurality of second pixels respectively corresponding to a plurality of regions into which the second visual field 231 is divided. For example, the second pixel may indicate a distance from the viewpoint in the viewing direction. Specifically, the pixel value of the second pixel may indicate, for example, a time difference from emission of electromagnetic waves to detection of reflected waves or scattered waves, or a phase difference between electromagnetic waves and reflected waves or scattered waves. The distance information may include, for example, a distance image in which each of the plurality of second pixels is color-coded according to the distance. The information generation device 20 may periodically generate distance information of the second visual field 231.

  The second information generation unit 23 may generate luminance information of the second visual field 231. The luminance information of the second visual field 231 may include a plurality of third pixels that respectively correspond to a plurality of regions into which the second visual field 231 is divided. The third pixel may indicate the luminance of the reflected wave or scattered wave of electromagnetic waves. Specifically, the pixel value of the third pixel may indicate the intensity of the reflected wave or the scattered wave. For example, the luminance information may include a luminance image in which each of the plurality of third pixels is color-coded according to luminance. The information generation device 20 may periodically generate luminance information of the second visual field 231.

  The communication unit 24 includes an arbitrary communication interface capable of communicating with an external device. The external device includes an information processing device 30, for example. The communication unit 24 may correspond to an arbitrary communication method via wired or wireless.

  The storage unit 25 may include one or more memories. In the present disclosure, the “memory” may include, for example, a semiconductor memory, a magnetic memory, an optical memory, and the like. The storage unit 25 may function as a primary storage device or a secondary storage device. The storage unit 25 stores arbitrary information and programs used for the operation of the information generation device 20.

  The control unit 26 includes one or more processors. The control unit 26 controls the overall operation of the information generation apparatus 20.

  For example, the control unit 26 may execute an arbitrary process using the imaging information. The process may include, for example, a viewpoint conversion process for imaging information. By the viewpoint conversion process, the viewpoint position and the optical axis 220 of the first information generation unit 22 can virtually coincide or completely coincide with the viewpoint position and the optical axis 230 of the second information generation unit 23, respectively. Any algorithm can be adopted for the viewpoint conversion process. The process using the imaging information is not limited to the viewpoint conversion process, and may include an arbitrary process. For example, the processing may include white balance processing, color adjustment processing, gain adjustment processing, luminance adjustment processing, and the like.

  The control unit 26 may output the imaging information, the distance information, and the luminance information to the information processing apparatus 30 via the communication unit 24, for example, periodically.

  The information processing apparatus 30 will be described with reference to FIG. The information processing apparatus 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

  The communication unit 31 includes an arbitrary communication interface that can communicate with an external device. The external device includes, for example, the information generation device 20. The communication unit 31 may correspond to an arbitrary communication method via wired or wireless.

  The storage unit 32 may include one or more memories. The storage unit 32 may function as a primary storage device or a secondary storage device. The storage unit 32 stores arbitrary information and programs used for the operation of the information processing apparatus 30.

  The control unit 33 includes one or more processors. The control unit 33 controls the overall operation of the information processing apparatus 30.

  For example, the control unit 33 acquires imaging information, distance information, and luminance information output by the information generation device 20 via the communication unit 31. The control unit 33 may execute an arbitrary process using the imaging information. For example, the control unit 33 may execute the above-described viewpoint conversion process or the like instead of the control unit 26 of the information generation device 20.

  The controller 33 executes a foreign object detection process based on the distance information. By the foreign object detection processing, for example, foreign objects attached on the surface region 210 of the optical member 21 shown in FIG. 2 can be detected.

  Specifically, when a foreign object adheres to the optical member 21, the electromagnetic wave emitted from the second information generation unit 23 of the information generation device 20 can be reflected or scattered by the foreign object. Therefore, in the distance information, the distance of the region on the second visual field 231 corresponding to the foreign object can be, for example, zero.

  The control unit 33 determines whether or not a specific region in the field of view whose distance is equal to or less than a predetermined value is detected in the distance information. The predetermined value may be zero, for example. When it is determined that a specific area within the field of view whose distance is equal to or less than the predetermined value is detected, the control unit 33 determines that a foreign object has been detected on the surface area 210 of the optical member 21. For example, FIG. 3 shows an example of the detected specific area 41. On the other hand, when it is determined that the specific area 41 within the visual field whose distance is equal to or smaller than the predetermined value is not detected, the control unit 33 determines that no foreign matter is detected on the surface area 210 of the optical member 21.

  If it is determined that a foreign object has been detected by the foreign object detection process described above, the control unit 33 may execute the foreign object classification process based on the distance information and the luminance information. By the foreign matter classification process, the detected foreign matter can be classified into, for example, water, muddy water, and mud.

  Specifically, the intensity of the reflected wave or scattered wave by water can be attenuated relatively greatly from the intensity of the emitted electromagnetic wave. The intensity of the reflected or scattered wave by the muddy water can be attenuated from the intensity of the emitted electromagnetic wave. The intensity of the reflected or scattered wave by the mud is hardly attenuated from the intensity of the emitted electromagnetic wave. As described above, the degree of attenuation of the intensity of the reflected wave or the scattered wave by the foreign matter varies depending on the ratio of the water contained in the foreign matter.

  The control unit 33 determines whether or not the luminance of the specific area 41 in the visual field whose distance is equal to or smaller than a predetermined value detected by the above-described foreign object detection processing is less than the first reference value. When the luminance of the specific area 41 is less than the first reference value, the control unit 33 classifies the foreign matter as water. On the other hand, when the luminance of the specific area 41 is equal to or higher than the first reference value, the control unit 33 determines whether the luminance of the specific area 41 is less than the second reference value that is larger than the first reference value. When the luminance of the specific area 41 is less than the second reference value, the control unit 33 classifies the foreign matter as muddy water. On the other hand, when the brightness | luminance of the specific area | region 41 is more than a 2nd reference value, the control part 33 classifies a foreign material into mud. The first reference value and the second reference value may be determined in advance based on experiments or simulations. The control unit 33 may output the execution result of the foreign object detection process and the foreign object classification process via the communication unit 31.

  The control unit 33 specifies a part of the first pixels included in the imaging information based on the distance information. Specifically, the control unit 33 corresponds to the specific region 41 in the field of view whose distance is equal to or smaller than a predetermined value, which is detected by the above-described foreign object detection process, among the plurality of first pixels included in the imaging information. One pixel is specified.

  The control unit 33 executes a predetermined process using the imaging information excluding the identified first pixel. The process may include an image recognition process, for example. Objects such as people and vehicles can be recognized by the image recognition processing. Any algorithm can be used for the image recognition processing. The control unit 33 may output the execution result of the process via the communication unit 31.

  The operation of the information processing system 10 will be described with reference to FIG. This operation may be repeatedly executed, for example, periodically.

  Step S100: The information generation device 20 generates imaging information of the first visual field 221, distance information of the second visual field 231, and luminance information of the second visual field 231.

  Step S101: The information generation device 20 outputs the generated imaging information, distance information, and luminance information to the information processing device 30.

  Step S102: The information processing apparatus 30 acquires imaging information, distance information, and luminance information.

  Step S103: The information processing apparatus 30 performs a foreign object detection process based on the distance information. It is determined whether or not a foreign object has been detected by the foreign object detection process. The operation of the information processing apparatus 30 that executes the foreign object detection process will be described later.

  Step S104: The information processing apparatus 30 executes the foreign substance classification process based on the distance information and the luminance information. The detected foreign matter is classified by the foreign matter classification process. The operation of the information processing apparatus 30 that executes the foreign substance classification process will be described later.

  Step S105: The information processing apparatus 30 specifies some first pixels included in the imaging information based on the distance information.

  Step S106: The information processing apparatus 30 executes a predetermined process using the imaging information excluding the identified first pixel.

  With reference to FIG. 5, the operation of the information processing apparatus 30 that executes the foreign object detection process and the foreign object classification process described above will be described.

  Step S200: The control unit 33 determines whether or not the specific area 41 in the visual field whose distance is equal to or smaller than a predetermined value is detected in the distance information. If it is determined that the specific area 41 is detected (step S200—Yes), the process proceeds to step S202. On the other hand, when it is determined that the specific area 41 is not detected (step S200-No), the process proceeds to step S201.

  Step S201: The controller 33 determines that no foreign matter is detected on the surface area 210 of the optical member 21.

  Step S202: The controller 33 determines that a foreign object has been detected on the surface region 210 of the optical member 21.

  Step S203: The control unit 33 determines whether or not the luminance of the specific area 41 detected in step S200 is less than the first reference value. When it is determined that the luminance is less than the first reference value (step S203—Yes), the process proceeds to step S204. On the other hand, when it is determined that the luminance is equal to or higher than the first reference value (step S203—No), the process proceeds to step S205.

  Step S204: The control unit 33 classifies the foreign matter detected in step S202 as water.

  Step S205: The control unit 33 determines whether or not the luminance of the specific area 41 detected in step S200 is less than a second reference value that is greater than the first reference value. If it is determined that the luminance is less than the second reference value (step S205—Yes), the process proceeds to step S206. On the other hand, when it is determined that the luminance is greater than or equal to the second reference value (step S205—No), the process proceeds to step S207.

  Step S206: The control unit 33 classifies the foreign matter detected in step S202 as muddy water.

  Step S207: The control unit 33 classifies the foreign matter detected in step S202 as mud.

  Said step S200-S202 is corresponded to a foreign material detection process. Steps S203 to S207 correspond to foreign object classification processing.

  As described above, the information processing apparatus 30 according to an embodiment acquires imaging information and distance information of a predetermined visual field. The information processing apparatus 30 specifies some first pixels among the plurality of first pixels included in the imaging information based on the distance information. The information processing apparatus 30 performs a predetermined process using the imaging information excluding the identified first pixel. According to this configuration, for example, the first pixel that is not appropriate for the processing can be excluded based on the distance information. For this reason, the precision of the execution result of the said process using imaging information can improve. Therefore, the convenience of the technique using the imaging information is improved.

  For example, when a foreign substance adheres on the surface region 210 of the optical member 21, the first pixel corresponding to the foreign substance in the imaging information is not necessarily appropriate in the above-described processing. On the other hand, the second pixel corresponding to the foreign object in the distance information has a high probability of indicating an extremely short distance. When the specific area 41 in the field of view whose distance is equal to or smaller than a predetermined value is detected based on the distance information, the information processing apparatus 30 sets the first pixel corresponding to the specific area 41 in the imaging information to the first portion described above. You may specify as a pixel. According to such a configuration, the first pixel corresponding to the foreign object can be excluded in the above-described processing. For this reason, the precision of the execution result of the said process using imaging information can improve. Therefore, the convenience of the technique using the imaging information is improved.

  Although the embodiments of the present disclosure have been described with reference to the drawings, it should be noted that those skilled in the art can easily make various variations or modifications based on the present disclosure. Accordingly, it should be noted that these variations or modifications are included in the scope of the present disclosure. For example, the functions included in each means or each step can be rearranged so that there is no logical contradiction. For example, a plurality of means or steps can be combined into one or divided.

  For example, each of the information generation device 20 and the information processing device 30 according to the embodiment described above may be divided into two or more devices. The information generation device 20 and the information processing device 30 may be integrated into one device.

  In the embodiment described above, the example in which the foreign matter is classified into three items of water, muddy water, and mud by the foreign matter classification process has been described. However, the contents of items to be classified and the number of items are not limited to the example described above. For example, the content and the number of items to be classified are arbitrarily determined based on a combination of an arbitrary first substance and an arbitrary second substance that attenuates the intensity of electromagnetic waves compared to the first substance. Also good.

  As the information processing apparatus 30 according to the above-described embodiment, any apparatus having an information processing function such as a general-purpose computer can be employed. For example, a program describing processing contents for realizing each function of the information processing apparatus 30 according to the embodiment is stored in the memory of the apparatus and is executed by the processor of the apparatus. Therefore, the embodiment of the present disclosure can also be realized as a program.

DESCRIPTION OF SYMBOLS 10 Information processing system 20 Information generation apparatus 21 Optical member 210 Surface area 22 1st information generation part 220 Optical axis 221 1st visual field 23 2nd information generation part 230 Optical axis 231 2nd visual field 24 Communication part 25 Storage part 26 Control part 30 Information processing apparatus 31 Communication unit 32 Storage unit 33 Control unit 40 Overlapping area 41 Specific area

Claims (11)

A communication unit for acquiring imaging information of a predetermined visual field and distance information of the visual field;
A control unit that identifies a part of the plurality of pixels included in the imaging information based on the distance information, and executes a predetermined process using the imaging information excluding the identified part of the pixel; ,
An information processing apparatus comprising: The information processing apparatus according to claim 1,
When the area within the field of view whose distance is equal to or less than a predetermined value is detected based on the distance information, the control unit sets pixels corresponding to the area within the field of view as the partial pixels in the imaging information. An information processing apparatus to be identified. The information processing apparatus according to claim 1, wherein:
The control unit determines that a foreign object has been detected when a region within the field of view whose distance is equal to or less than a predetermined value is detected based on the distance information. The information processing apparatus according to claim 3,
The control unit determines that the foreign matter is a foreign matter attached to an optical member shared by a first information generation unit that generates the imaging information and a second information generation unit that generates the distance information. Processing equipment. The information processing apparatus according to claim 4,
The information processing apparatus, wherein the optical member includes at least one of a housing window, a lens, a mirror, a filter, a cover, and glass. An information processing apparatus according to any one of claims 3 to 5,
The communication unit further acquires luminance information of the visual field,
The control unit is an information processing apparatus that classifies the foreign matter based on the distance information and the luminance information. The information processing apparatus according to claim 6,
The distance indicated in the distance information is determined based on a time difference from when an electromagnetic wave is emitted from a light source to when a reflected wave or scattered wave of the electromagnetic wave is detected,
The luminance indicated by the luminance information is an information processing apparatus that is determined based on the intensity of the reflected wave or the scattered wave of the electromagnetic wave. The information processing apparatus according to claim 7,
The information processing apparatus, wherein the electromagnetic wave includes infrared light. The information processing apparatus according to claim 8,
The electromagnetic wave includes information processing apparatus including near infrared light or far infrared light. In the information processing device,
Obtaining imaging information of a predetermined visual field and distance information of the visual field;
Identifying some of the plurality of pixels included in the imaging information based on the distance information;
Executing a predetermined process using the imaging information excluding the identified some pixels;
A program that executes An information generating device for generating imaging information of a predetermined visual field and distance information of the visual field;
An information processing apparatus that identifies a part of a plurality of pixels included in the imaging information based on the distance information and executes a predetermined process using the imaging information excluding the identified part of the pixel When,
An information processing system comprising:

Images