JP6587253B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Conventionally, a number of systems using cameras have been proposed in order to grasp the movements of people and surrounding conditions in various applications such as stores, houses, and cars. In such a system, there is a demand for monitoring a specific area in the camera video in detail according to the scene. Especially for passenger vehicles such as buses and trains, it is possible to monitor the vicinity of the platform and the entrance / exit for various reasons such as preventing accidents at the platform and operating safely or detecting the degree of congestion of passengers at the platform. A system is needed.

In order to meet such a demand, Patent Document 1 discloses an imaging system that monitors a subject having a depth such as a train from a camera installed on the home side from an oblique direction, and displays a subject having a perspective from the front. Is disclosed. As a result, it is possible to correct the change in the size of a passenger or the like reflected in the video depending on the distance from the camera, thereby improving the visibility.
Further, Patent Document 2 discloses an image processing apparatus that generates a plurality of partial images indicating regions near the entrance / exit using a camera capable of photographing a plurality of entrances / exits in a passenger vehicle, and displays the partial images side by side. It is disclosed. As a result, it is possible to appropriately perform safety confirmation near the entrance / exit with a small number of cameras.
Similarly, in Patent Document 3, a plurality of partial images showing regions near the entrance / exit are generated using a camera capable of photographing a plurality of entrances / exits in the passenger vehicle, and the vicinity of the entrance / exit where the object is detected An image processing apparatus that displays only the image on the display device is disclosed. This makes it possible for the crew members to pay attention only to the entrance / exit that requires safety confirmation.

JP 2006-033418 A JP 2012-054664 A JP 2012-134825 A

As described above, various systems have been proposed for the purpose of monitoring homes and passengers. For example, by installing a camera on the passenger vehicle side, it is possible to monitor homes and passengers even in an environment where it is difficult to install a camera such as an unmanned station.
However, in the imaging system described in Patent Document 1, it is possible to improve the visibility in the image of the camera installed in the home, but the visibility of the image of the camera installed in the passenger vehicle side such as a train is possible. There is no mention of improvements.
Moreover, in the image processing apparatus described in Patent Document 2, a camera capable of photographing a plurality of entrances and exits in a passenger vehicle is used, and a plurality of partial images showing regions near the entrances and exits are displayed side by side. Although it is possible to ensure safety near the entrance / exit, it is not possible to grasp the entire home because it is an image only near the entrance / exit, for example, it is delayed to visually recognize passengers who are going to get on board Sometimes.
In addition, the image processing apparatus described in Patent Document 3 can detect an object in addition to generating a partial image in the vicinity of the entrance / exit, so that attention can be paid only to the entrance / exit where attention should be paid. Similarly, the outside of the partial image cannot be visually recognized, and the visual recognition may be delayed with respect to a sudden state change near the entrance / exit.
Thus, conventionally, it has been difficult to obtain a captured image with good visibility over a wide range.

  The present invention has been made in view of the above points, and relates to an image processing apparatus and an image processing method capable of obtaining a captured image with high visibility over a wide range.

  The present invention has been made to solve the above problems, and one aspect of the present invention is an image input unit that inputs an image captured by a camera and an image in which a reference object on a specific surface is captured. When a predetermined region of interest exists in the periphery of the image, the image for correcting the image so that an angle on the image of the reference object with respect to the specific surface within the region of interest falls within a predetermined angle. An image processing apparatus comprising: a correction amount determination unit that determines a correction amount; and an image correction unit that corrects an image input by the image input unit based on the correction amount determined by the correction amount determination unit. .

  According to another aspect of the present invention, in the image processing apparatus, the correction amount determination unit is configured such that a width of the reference object on the specific surface on the specific surface is greater than or equal to a predetermined width in the attention area. The correction amount is further determined.

  In one aspect of the present invention, the image processing apparatus further includes a risk determination unit that determines a risk based on the input image, and the correction amount determination unit is determined by the risk determination unit. The correction amount is determined according to the degree of risk to be performed.

  According to another aspect of the present invention, the image processing apparatus further includes a correction auxiliary information setting unit that sets correction auxiliary information used when the correction amount is determined, and the correction amount determination unit includes the correction amount determination unit. The correction amount is determined based on the correction auxiliary information set by the auxiliary information setting unit.

  According to another aspect of the present invention, in the information image processing apparatus, when the correction amount determination unit determines the correction amount, an area excluding a part of the image area of the image is enlarged. The correction amount is determined as follows.

  Further, according to one embodiment of the present invention, in the image input step of inputting an image captured by the camera and the image of the reference object on the specific surface, a predetermined attention area exists in the peripheral portion of the image A correction amount determining step for determining a correction amount for correcting the image so that an angle of the reference object with respect to the specific surface on the image within a predetermined angle in the region of interest falls within a predetermined angle; and the correction amount determination And an image correction step of correcting the image input in the image input step based on the correction amount determined in the step.

  According to the present invention, it is possible to obtain a captured image with good visibility over a wide range.

1 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to a first embodiment. The figure which shows the installation environment example of the wide angle camera in 1st Embodiment. The figure which shows the example of an image image | photographed by the wide angle camera in 1st Embodiment. 6 is a flowchart illustrating an example of image correction processing according to the first embodiment. The figure which shows an example of the picked-up image before correction | amendment in 1st Embodiment. The figure explaining the correction amount in 1st Embodiment. The figure which shows an example of the correction | amendment image after correction | amendment in 1st Embodiment. The block diagram which shows an example of schematic structure of the image processing apparatus in 2nd Embodiment. 9 is a flowchart illustrating an example of image correction processing according to the second embodiment. The figure explaining determination of the risk in the picked-up image in 2nd Embodiment. The figure which shows an example of the correction | amendment image after correction | amendment in 2nd Embodiment. FIG. 10 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to a third embodiment. 10 is a flowchart illustrating an example of image correction processing according to the third embodiment. The figure which shows an example of the attention area | region in 3rd Embodiment. The figure explaining the corrected amount in 3rd Embodiment. The figure which shows the installation environment example of the wide angle camera in 3rd Embodiment. FIG. 10 is a diagram illustrating an application example of an image processing apparatus according to a fourth embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted that the size of each part in each drawing is exaggerated in terms of the magnitude relationship for ease of explanation, and is different from the actual size.

(First embodiment)
FIG. 1 is a block diagram showing an example of a schematic configuration of an image processing apparatus 100 according to the first embodiment of the present invention. The image processing apparatus 100 sets camera position information such as an attachment position and an angle of the wide angle camera 106 based on an image input unit 101 that inputs a captured image captured by the wide angle camera 106 and parameters set by the user. When correcting the lens data of the camera position setting unit 102, the wide-angle camera 106, the storage unit 103 that stores the camera position information set by the camera position setting unit 102, and the captured image input to the image input unit 101 A correction amount determination unit 104 that determines a correction amount and an image correction unit 105 that corrects a captured image based on the correction amount determined by the correction amount determination unit 104 are configured. The image processing apparatus 100 is connected to a wide-angle camera 106 that outputs a captured image to the image processing apparatus 100 and an image display unit 107 that displays a video output output from the image processing apparatus 100.

  The image processing apparatus 100 includes semiconductor components such as a central processing unit called a CPU (Central Processing Unit), a field-programmable gate array (FPGA), and an application specific integrated circuit (ASIC) called an ASIC (Application Specific Integrated Circuit). 1 is configured by combining a storage medium such as a hard disk, flash memory, or DRAM (Dynamic Random Access Memory), and implements the functions shown in FIG.

  The wide-angle camera 106 is composed of an image sensor such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS) and a plurality of components including a lens with a wide angle of view (wide angle lens). Details of the lens and the angle of view will be described later. The image display unit 107 is a display device including a display device such as liquid crystal or organic EL (Electro-Luminescence).

  FIG. 2 is a diagram showing an installation environment example of the wide-angle camera 106 in the present embodiment. A wide-angle camera 106 is attached near the center of the passenger vehicle 108, and the passenger vehicle 108 has three entrances 109, 110, and 111. It is assumed that the wide-angle camera 106 is attached obliquely downward with respect to the side surface of the passenger vehicle 108 so that the optical axis direction faces a position away from the entrance 110 by a certain distance. That is, the mounting angle of the wide-angle camera 106 is an acute angle with respect to the side surface of the passenger vehicle 108 (the surface on which the wide-angle camera 106 is mounted). Moreover, as shown in this figure, the wide-angle camera 106 is installed above the entrance 110 of the passenger vehicle 108 or at a higher position. By installing the wide-angle camera 106 in this way, in addition to the vicinity of the entrances 109, 110, and 111, a home area that is desirably visible can be widely included in the image. In addition, even when there are passengers in the vicinity, it is possible to take a picture of the vicinity of a plurality of entrances and exits without being blocked. In order to photograph a plurality of entrances / exits as much as possible, it is desirable to install in the vicinity of the center in the longitudinal direction of the side surface of the passenger vehicle 108 as shown in FIG. The image processing apparatus 100 is installed inside the passenger vehicle 108, for example.

  FIG. 3 is a view showing an example of an image taken by the wide-angle camera 106 attached to the passenger vehicle 108. The wide-angle camera 106 is, for example, a fish-eye camera equipped with a fish-eye lens, and has a wide angle of view of approximately 170 degrees in the vertical and horizontal directions. The passenger vehicle 108 is photographed in the photographed image 112 shown in the figure. As can be seen from the photographed image 112 that all of the entrances 109, 110, and 111 in the passenger vehicle 108 are shown, a wide area is photographed, and the distortion exists in the lens. In response, the passenger vehicle 108 is photographed while being deformed from its original shape. A black line 113 indicates a boundary (gap) portion between the home and the vehicle. Note that the black portions at the four corners of the photographed image 112 are portions where no video is photographed because they are outside the lens field of view. The image processing apparatus 100 receives the captured image 112 by the wide-angle camera 106 as input, and corrects the input captured image 112 so that a wide range including the home can be visually recognized, and passengers near the entrance / exit, that is, the passenger vehicle 108. A corrected image that can improve the visibility of a person who enters and exits is generated.

FIG. 4 is a flowchart illustrating an example of image correction processing executed by the image processing apparatus 100. Hereinafter, the operation of the image correction processing in the present embodiment will be described with reference to FIG.
First, a captured image captured by the wide-angle camera 106 is input to the image input unit 101 (step S10). The image input unit 101 sends the input captured image to the image correction unit 105. Note that, here, a configuration in which a captured image is directly input to the image input unit 101 from the wide-angle camera 106 is taken as an example, but the camera and the image processing apparatus 100 are connected by wired or wireless communication, so that the camera The captured image may be input directly to the image input unit 101, or the captured image may be input to the image input unit 101 via a storage medium such as a memory card. A captured image stored in a server device (not shown) installed outside the image processing apparatus 100 or a captured image stored in the storage unit 103 inside the image processing apparatus 100 is input to the image input unit 101. It may be entered.

  Further, the correction amount determination unit 104 acquires camera position information of the wide-angle camera 106 from the storage unit 103 (step S11). The correction amount determination unit 104 uses this camera position information when determining the correction amount used when the image correction unit 105 corrects an image.

  The above-described camera position information is set by the camera position setting unit 102. For example, the user inputs camera position information or information for determining camera position information as a parameter from an operation unit (not shown). The camera position setting unit 102 sets camera position information such as the attachment position and angle of the wide-angle camera 106 based on the input parameters. The camera position information may be read from a data file stored in the storage medium. For example, the camera position information includes not only the height from the ground with respect to the reference point, but also the angle in the optical axis direction of the wide-angle camera 106 with respect to the left, right, and top and bottom. The camera position setting unit 102 accepts values obtained by actual measurement of various camera position information by the user with respect to the attached wide-angle camera 106 and stores the values in the storage unit 103.

  Further, the correction amount determination unit 104 acquires vehicle information stored in advance in the storage unit 103 in addition to acquiring camera position information (step S12). The vehicle information is information related to the structure of the vehicle, for example, information including the total length of the passenger vehicle 108 shown in FIG. 2, position information on the vehicles at the entrances 109, 110, and 111. The correction amount determination unit 104 also acquires lens information of the wide-angle camera 106 from the storage unit 103 (step S13). The lens information is information relating to optical distortion of the lens, and is information indicating at which position on the image sensor an object actually photographed is imaged. The correction amount determination unit 104 is a captured image captured by the wide-angle camera 10 based on the camera position information of the wide-angle camera 106 acquired from the storage unit 103, the lens information of the wide-angle camera 106, and the vehicle information of the passenger vehicle 108. Is determined (step S14).

Here, the correction amount determined by the correction amount determination unit 104 and the determination method will be described with reference to FIGS.
FIG. 5 is a diagram illustrating an example of a captured image captured by the wide-angle camera 106 before correction. In the illustrated captured image 114, the passenger vehicle 108 is shown, and the entrances 109, 110, and 111 are visible. In addition, passengers 115, 116, and 117 standing at the platform in front of the entrances 109, 110, and 111 are photographed. Here, the passenger 116 is reflected from the center of the captured image 114, but the passengers 115 and 117 are reflected in the peripheral portion of the captured image 114. Therefore, the passengers 115 and 117 appear to be inclined with respect to the vertical direction of the captured image 114 due to the influence of optical distortion of the lens. Here, the vertical direction of the captured image 114 is a direction perpendicular to the base 114a of the captured image 114 shown in the figure.

  In each of the passengers 115, 116, and 117 standing in front of each entrance, center lines A, B, and C indicated by broken lines are lines connecting the center between each passenger's face center to both feet, A line perpendicular to (home). Comparing the center lines A, B, and C, the center line B is parallel to the vertical direction of the captured image 114 (same direction), but the center lines A and C are inclined with respect to the vertical direction of the captured image 114, The angle on the captured image 114 is different. That is, the passenger 116 is shown in a straight state in the same direction as the vertical direction of the captured image 114, but the passengers 115 and 117 are displayed in a state inclined with respect to the vertical direction of the captured image 114. Here, it is assumed that the widths of the bodies of the passengers 115, 116, and 117 are substantially the same, but the widths W1, W2, and W3 of the passengers 115, 116, and 117 shown in the captured image 114 are In comparison, the widths W1 and W3 are narrower than the width W2 of the passenger 116.

  The captured image 114 is such an image due to the characteristics of the fisheye lens used in the wide-angle camera 106. The image processing apparatus 100 determines a correction amount for the captured image 114 and corrects the image. Here, the area around the entrances 109, 110, and 111 is defined as the attention area. Based on the camera position information of the wide-angle camera 106 acquired from the storage unit 103, the lens information of the wide-angle camera 106, and the vehicle information of the passenger vehicle 108, which position on the image captured by the wide-angle camera 106 is at each entrance / exit. You can see if it is nearby. The correction amount determination unit 104 selects from among a plurality of predetermined correction amounts held in the storage unit 103 according to the position on the image near the entrance / exit, which is the region of interest most desired to be monitored by the wide-angle camera 106. Determine the optimal correction amount.

  FIG. 6 is a diagram illustrating a correction amount for correcting the influence of the optical distortion of the lens. This figure is an example of a plurality of correction amounts, and shows an example of correction amounts for the captured image 114. A thick frame indicated by reference numeral (a) indicates a captured image 114 before correction. On the other hand, a thick frame indicated by reference numeral (b) indicates a corrected image 118 after the captured image 114 is corrected. The coordinates of each black point shown as a representative point on the photographed image 114 shown by the symbol (a) are converted into the coordinates of each black point shown on the corrected image 118 shown by the symbol (b) by correction. Some of the representative black dots are shown with arrows so that the correspondence can be easily understood.

  Here, information regarding each black spot on the photographed image 114 before correction corresponding to each black spot on the corrected image 118 is defined as a correction amount. Based on the correspondence relationship between the coordinates before and after the conversion at the representative point indicated by the black point, for the coordinates other than the black point, the corresponding coordinates are calculated and converted by a complementary process such as linear interpolation. Here, the table has coordinates on the photographed image 114 corresponding to the coordinates of each representative black point predetermined on the corrected image 118, but the number of representative points (black points) shown here is not limited. More or less, or a table indicating coordinates on the photographed image 114 corresponding to all coordinates of the corrected image 118 may be used. The amount of data increases as the number of representative points increases, but the corrected image is corrected to an image more in line with the intention.

  Returning to FIG. 4, the image correction unit 105 corrects the captured image 114 using the correction amount determined by the correction amount determination unit 104 (for example, the correction amount described with reference to FIG. 6) (step S15). ). For example, the image correcting unit 105 converts the pixel value of the coordinates of each black point on the corrected image 118 shown in FIG. 6 into the pixel value of the coordinates of the photographed image 114 before correction corresponding to each black point. Further, the image correction unit 105 converts each coordinate other than the black point on the corrected image 118 to a pixel value of each coordinate of the photographed image 114 before correction calculated by a complementing process such as linear interpolation. Then, the image correction unit 105 outputs the corrected image 118 corrected to the image display unit 107 (step S16). As a result, the corrected image 118 is displayed on the image display unit 107.

  FIG. 7 is a diagram illustrating an example of the corrected image 118 after correction. The center lines A, B, and C of the passengers 115, 116, and 117 shown in the photographed image 114 before correction shown in FIG. 5 are the center lines A ′, B ′, and C in the corrected image 118 after correction shown in FIG. Corresponds to '. The center line B before correction and the center line B ′ after correction do not change before and after the correction, and are parallel to the vertical direction of the image (the photographed image 114 and the corrected image 118). On the other hand, when comparing the center line A before correction with the center line A ′ after correction, the center line A ′ after correction is closer to the vertical direction of the image (corrected image 118) than the center line A before correction. It is corrected to an angle. Similarly, the relationship between the center line C before correction and the center line C ′ after correction is similar to the center line C ′ after correction closer to the vertical direction of the image (corrected image 118) than the center line C before correction. It is corrected to an angle. That is, in the attention area at the periphery of the captured image 114 before correction, the angle on the image of the passengers 115 and 117 that is perpendicular to the ground (the top surface of the home) is corrected to be within a predetermined angle. . Here, “within a predetermined angle” means that, for example, the angle on the image of the passenger 116 reflected in the vicinity of the center of the captured image 114 (that is, the angle with respect to the vertical direction on the image) is a predetermined angle (± 10 degrees, ± 20 degrees).

  Further, the widths W1, W2, and W3 of the passengers 115, 116, and 117 shown in the photographed image 114 before correction shown in FIG. 5 are the widths W1 ′, W2 ′, and W3 in the corrected image 118 after correction shown in FIG. Corresponds to '. The corrected widths W1 'and W3' of the passengers 115 and 117 are wider than the uncorrected widths W1 and W3, and the passengers 115 and 117 are corrected to an image that is easier to visually recognize. That is, in the attention area in the periphery of the captured image 114 before correction, the width on the image of the passengers 115 and 117 that is perpendicular to the ground (the top surface of the home) is larger than a predetermined width. It is corrected. Here, the predetermined width is, for example, smaller than the width W2 of the passenger 116 reflected in the vicinity of the center of the photographed image 114 before correction, and larger than the widths W1 and W3 of the passengers 115 and 117 reflected in the peripheral portion. Width. By correcting the optical distortion of the lens, the corrected width W2 'of the passenger 116 is narrower than the width W2. However, since the passenger 116 is originally reflected in the center of the image and is large in size, the visibility is not deteriorated so much.

  As described above, the image processing apparatus 100 according to this embodiment includes the image input unit 101, the correction amount determination unit 104, and the image correction unit 105. For example, the image input unit 101 inputs an image taken by a camera. In a captured image obtained by capturing a reference object (for example, a passenger) on a specific surface (for example, the ground or a home), the correction amount determining unit 104 is configured so that a predetermined attention area exists in the peripheral portion of the captured image. A correction amount for correcting the image is determined so that an angle (for example, vertical) of the reference object with respect to the specific surface on the captured image falls within a predetermined angle in the attention area. The image correction unit 105 corrects the captured image input by the image input unit 101 based on the correction amount determined by the correction amount determination unit 104.

As a result, the image correction unit 105 can obtain a captured image with good visibility over a wide range. For example, in the image processing apparatus 100, a passenger standing perpendicular to the ground (home) in a region of interest in the periphery of a captured image captured by a wide-angle camera 106 installed in a passenger vehicle 108 has an optical distortion of the lens. When the image is tilted due to the influence, the captured image can be corrected so that the image can be visually recognized at a closer angle.
The above-mentioned reference object is an object having a predetermined size in a direction perpendicular to a specific surface (for example, the ground, a home, etc.), and may be a person or an object such as a stick or a box. There may be.

  Further, the correction amount determination unit 104 performs the above correction so that the width of the reference image (for example, passenger) on the specific surface (for example, the ground or the platform) on the captured image is greater than or equal to a predetermined width in the attention area. The amount may be further determined. As a result, the image processing apparatus 100 is more practical when the passenger appears smaller due to the optical distortion of the lens in the attention area around the captured image captured by the wide-angle camera 106 installed in the passenger vehicle 108. The photographed image can be corrected so that it can be viewed with a size close to.

  As described above, the image processing apparatus 100 according to the present embodiment is, for example, perpendicular to the ground (home) in the attention area around the captured image captured by the wide-angle camera 106 installed in the passenger vehicle 108 instead of the home. When a standing passenger or object is slightly tilted due to the optical distortion of the lens, it can be corrected so that it can be seen at an angle or size that is closer to reality. Therefore, according to the image processing apparatus 100 of the present embodiment, it is possible to obtain a captured image in which a wide range is visible and the visibility of a specific region (region of interest) is improved.

For example, as described with reference to FIG. 6, a plurality of correction amounts are prepared in accordance with the position of each region of interest in the captured image, so that each region of interest in the captured image is perpendicular to the ground. The reference object is set such that the angle on the image of the reference object falls within a predetermined angle set in advance and the width of the reference object is equal to or larger than a predetermined size.
Therefore, for example, the image processing apparatus 100 is a captured image with a small blind spot capable of visually recognizing not only the attention area but also all the areas around the passenger vehicle 108, and the visibility in the attention area reflected in the periphery of the captured image. Can be increased. Therefore, when viewed as the entire shooting range of the input shot image, the image processing apparatus 100 can provide an image with less visibility and higher visibility than before correction.

  In the present embodiment, the correction amount is set such that the amount of correction increases as the distance from the vicinity of the entrance / exit 110 in the center of the captured image 114 to the entrances 109 and 111 in the periphery increases. Although described, the entrance / exit is not always shown at a position near the periphery of the captured image 114. From among a plurality of correction amounts, it is desirable to determine the correction amount so that the vicinity of the entrance / exit is most visible with reference to the position of the entrance / exit.

  In addition, the fish-eye lens used in the wide-angle camera 106 has several projection methods generally called equidistant projection or stereoscopic projection so that the resolution of the outer periphery of the lens is higher in stereoscopic projection. Designed. The widths W1, W2, and W3 shown in FIG. 5 vary depending on the projection method, but the magnitude relationship does not change, and the image correction process according to the present embodiment can be applied to any projection method.

In the present embodiment, as described with reference to FIG. 6, an example in which a plurality of correction amounts are provided according to the position in the captured image has been described, but depending on the installation information of the wide-angle camera 106 and the position of the attention area, The correction amount may be calculated based on the lens data of the wide angle camera 106. The lens of the wide-angle camera 106 has a distortion characteristic formula determined according to the projection method of the lens. For example, in the case of stereoscopic projection, the image height y, the focal length f, and the incident angle θ [radian] according to the angle from the lens center. ] Is represented by the following formula.
y = 2f × tan (θ / 2)

  The image processing apparatus 100 may determine how to shoot the reference object in each region of interest using the characteristic equation as described above and determine the correction amount each time. In addition to the camera installation height, orientation, angle, etc., if the vehicle information (position of the area of interest) is known, it is possible to know which angle of incidence each point in that area will have, so such image correction processing Is possible. In the case of this processing, calculation is required when the wide-angle camera 106 is installed. However, since it is not necessary to store the correction amount as a table in advance, there is an effect of reducing the memory capacity required for the processing and improving the degree of freedom in correction. is there.

  In the present embodiment, the image processing apparatus 100 is described as being installed in the passenger vehicle 108. However, the image processing apparatus 100 may be installed outside the passenger vehicle 108. For example, the wireless communication unit may be connected to the wide-angle camera 106 and the image processing. Each of the devices 100 may be configured to be connected by wireless communication in accordance with a wireless communication standard.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
FIG. 8 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 200 according to the present embodiment. In this figure, components having the same functions as those in the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image processing apparatus 200 includes an image input unit 101, an object detection unit 201, a camera position setting unit 102, a storage unit 103, a risk determination unit 202, a correction amount determination unit 203, and an image correction unit 105. The object detection unit 201, the risk determination unit 202, and the correction amount determination unit 203 are different from the image processing apparatus 100 illustrated in FIG.

  The object detection unit 201 analyzes the captured image input to the image input unit 101 and detects an object shown in the captured image. For example, the object detection unit 201 sends object detection information including the image and the coordinates of the detected object to the risk determination unit 202 for each frame of the captured image. The risk determination unit 202 determines the risk according to the state of the object detected by the object detection unit 201. For example, the risk determination unit 202 sets a risk determination area based on the object detection information sent from the object detection unit 201 and the camera position information and the vehicle information stored in the storage unit 103 to set the risk. Determine the degree. The correction amount determination unit 203 determines a correction amount for correcting the captured image according to the risk level determined by the risk level determination unit 202. The image processing apparatus 200 is connected to a wide-angle camera 106 that outputs a captured image to the image processing apparatus 200 and an image display unit 107 that displays a video output from the image processing apparatus 200.

  FIG. 9 is a flowchart illustrating an example of image correction processing executed by the image processing apparatus 200. Hereinafter, with reference to FIG. 9, the operation of the image correction processing in the present embodiment will be described.

  First, a captured image captured by the wide-angle camera 106 is input to the image input unit 101 (step S20). The image input unit 101 sends the input image to the object detection unit 201 and the image correction unit 105.

  The object detection unit 201 analyzes the captured image and detects an object shown in the captured image (step S21). Various methods are known as object detection algorithms. In addition to background difference and interframe difference methods that detect moving objects from temporal image difference data, pattern matching methods and HOG (Histograms of Oriented) (Gradients) There are various methods such as a human detection method using feature amounts and a learning method using deep learning which is a kind of neural network.

  Next, the object detection unit 201 sends object detection information including the video and the coordinates of the detected object to the risk determination unit 202 for each frame of the captured image. The risk determination unit 202 acquires the object detection information sent from the object detection unit 201, acquires camera position information and vehicle information from the storage unit 103, sets a risk determination area, and determines the risk. (Steps S22, S23, S24).

  FIG. 10 is a diagram for explaining detection of an object in a captured image and determination of the degree of risk. The photographed image 204 shown in the figure shows a passenger 205 and a passenger 206, and the broken line 205a around the passenger 205 and the broken line 206a around the passenger 206 indicate that an object is detected as a moving object. An area sandwiched between a black line 113 indicating the boundary between the home and the passenger vehicle 108 and a black line 207 indicating a predetermined fixed distance from the boundary is defined as a risk determination area. The risk determination area is set based on camera position information and vehicle information set in advance by the user via the camera position setting unit 102, and is determined in advance from a range of a certain distance from the passenger vehicle 108, that is, the black line 113. Range. Information regarding the actual distance indicating the certain distance range is held in the storage unit 103 and the risk determination unit 202. The risk determination unit 202 calculates the position of the object and the speed of movement based on the detection information received from the object detection unit 201, and determines whether or not the vehicle has entered the risk determination area. The speed here is the actual moving speed of the object or the position change amount on the image per predetermined unit time. Then, the risk determination unit 202 determines that the object detected by the object detection unit 201 has entered the risk determination area, and if the moving speed of the object is equal to or higher than a certain speed, Judgment is made and the risk is determined to be high (high risk). In other cases, the risk determination unit 202 determines that the risk is low (the risk is low). When it is determined that the degree of risk is high, the area where the object exists is the attention area, and the captured image is corrected so that the object can be easily noticed. Here, the correction that is easy to focus on is, for example, correction that increases the width and size of the passenger.

  For example, the risk determination unit 202 determines whether or not the detected object has a high risk (step S25). If it is determined that the risk is high (step S25: YES), the information and the high risk are determined. Is sent to the correction amount determination unit 203. The correction amount determination unit 203 determines the correction amount of the captured image so that it is easy to focus on a region including an object determined to have a high degree of risk (step S26). That is, the correction amount is determined so that the passenger's width is larger than that of the other areas, or the angle of the passenger's center line with respect to the image is closer to the vertical direction with respect to the high risk area. On the other hand, when it is determined that the risk level is low (step S25: NO), the risk level determination unit 202 is based on the camera position information, the lens information, and the vehicle information of the passenger vehicle 108, as in the first embodiment. Then, the correction amount of the photographed image is determined (step S27).

  The image correction unit 105 corrects the captured image based on the correction amount determined by the correction amount determination unit 203 (step S28). Then, the image correction unit 105 outputs the corrected image that has been corrected to the image display unit 107 (step S29). As a result, the corrected image is displayed on the image display unit 107.

  FIG. 11 is a diagram illustrating an example of a corrected image after correction by the image correction processing in the present embodiment. In the corrected image 208 shown in the figure, it is assumed that the passenger 205 is about to rush and get on and it is determined that the risk level is high. Center lines A and B indicated by broken lines in each of the passengers 205 and 206 indicate directions perpendicular to the ground (home), and are corrected to an angle closer to the vertical direction on the image than before correction. ing. Further, when comparing the width of the body on the images of the passengers 205 and 206, the width W1 of the passenger 205 is corrected so as to be wider than the width W2 of the passenger 206, so that the high-risk portion is more visually recognized. It has become easier.

As described above, the image processing apparatus 200 according to the present embodiment includes the risk determination unit 202 that determines the risk based on the input captured image, and the risk determined by the risk determination unit 202. The correction amount for correcting the captured image is determined according to the above.
As a result, the image processing apparatus 200 can correct the captured image so that an object with a high degree of risk can be more visually recognized. For example, the image processing apparatus 200 detects an object in the vicinity of the passenger vehicle 108 in the captured image, determines the risk level, and changes the correction amount according to the determined risk level, thereby widening the periphery of the passenger vehicle 108. While making it possible to monitor the range with an easy-to-view image, it is particularly easy to visually recognize and pay attention to a dangerous area. Therefore, the monitor around the passenger vehicle 108 can quickly and accurately grasp the dangerous situation.

(Third embodiment)
Next, a third embodiment of the present invention will be described.
FIG. 12 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 300 according to the present embodiment. In this figure, components having the same functions as those in the first and second embodiments shown in FIGS. 1 and 8 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image processing apparatus 300 includes an image input unit 101, an attention area setting unit 301, a storage unit 103, a correction auxiliary information setting unit 302, a correction amount determination unit 303, and an image correction unit 105. The image processing apparatus 100 shown in FIG. 1 and the image processing apparatus 200 shown in FIG. 8 are different in the configuration of the attention area setting unit 301, the correction auxiliary information setting unit 302, and the correction amount determination unit 303.

  The attention area setting unit 301 sets an attention area in accordance with an instruction from the user, and causes the storage unit 103 to store information regarding the attention area. The correction auxiliary information setting unit 302 sets correction auxiliary information used when determining the correction amount at the time of correction, and stores the correction auxiliary information in the storage unit 103. Details of the auxiliary correction information will be described later. The correction amount determination unit 303 determines the correction amount for correcting the captured image based on the information regarding the attention region set by the attention region setting unit 301 and the correction auxiliary information set by the correction auxiliary information setting unit 302. decide. The image processing apparatus 300 is connected to a wide-angle camera 106 that outputs a captured image to the image processing apparatus 300 and an image display unit 107 that displays a video output from the image processing apparatus 300.

  FIG. 13 is a flowchart illustrating an example of image correction processing executed by the image processing apparatus 300. Hereinafter, the operation of the image correction processing in the present embodiment will be described with reference to FIG.

  First, a captured image captured by the wide-angle camera 106 is input to the image input unit 101 (step S30). The image input unit 101 sends the input image to the image correction unit 105. The correction amount determination unit 303 acquires information on the attention area and correction auxiliary information from the storage unit 103 (steps S31 and S32).

  Here, the attention area will be described. The attention area is a position on the image while the user who monitors the conductor of the passenger vehicle 108, for example, looks at the image display unit 107 based on the captured image captured by the wide-angle camera 106 installed on the passenger vehicle 108. And set through the attention area setting unit 301. FIG. 14 is a diagram illustrating an example of an attention area set by a user in a captured image. In the photographed image 304 shown in the figure, areas 305, 306, and 307 indicated by diagonal lines are attention areas. Here, the vicinity of the entrance / exit is set as the attention area.

  In this way, as described in the first and second embodiments, in addition to the method of identifying the vicinity of the entrance / exit as the attention area based on the vehicle information and determining the correction amount, from the captured image captured by the wide-angle camera 106. It is also possible to specify a region of interest and determine the correction amount. In this way, it is possible to specify the region of interest without measuring and storing in advance information such as the attachment position of the wide-angle camera 106 to the passenger vehicle 108.

  Next, correction auxiliary information will be described. The correction auxiliary information is information that is referred to when determining the correction amount, for example, operation information of the passenger vehicle 108. The operation information is information for specifying the current situation of the passenger vehicle 108. For example, the operation information is information such as “running”, “entrance / exit closure”, “entrance / exit opening”, “before / after entrance / exit”. “Running” is defined as a situation in which the passenger vehicle 108 is traveling normally (traveling at a predetermined speed or higher). “Entrance / gate closure” is defined as a situation where the entrance / exit is closed. “Opening / exiting” is defined as a situation where passengers get on and off at the platform or stop of a station. “Before and after opening / closing the entrance / exit” means that the passenger vehicle 108 stops at the platform of the station, etc., until the entrance / exit is opened after the stop or until a certain time before the entrance / exit is closed. Define until it reaches.

  In addition, the definition of these operation information is an example, and what kind of information may be sufficient if it is information relevant to a driving | running | working state, the opening / closing state of a boarding / alighting gate, etc. Further, the correction auxiliary information is not limited to the operation information, and may be other information. For example, the correction auxiliary information includes information on whether or not it is a commuting time zone, information on whether it is a weekday, weekend or public holiday, simply time or time zone information, season information, the number of trained vehicles of the passenger vehicle 108, Information such as vehicle specifications may be included.

  The correction auxiliary information setting unit 302 stores operation information in the storage unit 103 in association with time according to an instruction from the user in advance. The correction auxiliary information setting unit 302 may determine operation information based on the state of various sensors mounted on the passenger vehicle 108 and may store the operation information in the storage unit 103.

  The correction amount determination unit 303 determines the correction amount of the captured image based on the above-described attention area and operation information (step S33 in FIG. 13). For example, the storage unit 103 corrects the attention area so that the angle on the image of the reference object standing perpendicular to the ground in each area in the captured image is within a predetermined angle and is greater than or equal to a predetermined size. A plurality of correction tables in which amounts are defined are held in advance, and the correction amount determination unit 303 extracts a plurality of correction tables held in advance in the storage unit 103. Furthermore, the correction amount determination unit 303 selects one correction table from a plurality of correction tables based on the operation information. For example, when the operation information is “open / exit opening”, it is important that the vicinity of the entrance / exit is clearly visible in order to check whether the passenger is getting on / off safely. Therefore, when the operation information is “opening / exit opening”, for example, a correction amount as shown in FIG. 15 is set.

  FIG. 15 is a diagram for explaining the correction amount in the present embodiment. The captured image 304 indicated by reference numeral (a) and the corrected image 308 indicated by reference numeral (b) are images before and after correction as in FIG. 6 described above, and each black dot indicates a correspondence relationship before and after correction. Unlike the example of the photographed image 114 shown in FIG. 6, the corresponding black spot does not exist in the region indicated by the symbol L at the top of the photographed image 304. For this reason, in the corrected image 308 after correction, the upper part of the photographed image 304 before correction is deleted.

  FIG. 16 is a diagram illustrating an installation environment example of the wide-angle camera 106 in the passenger vehicle 108, as viewed from the front of the passenger vehicle 108. An angle 106c formed by a broken line 106a and a broken line 106b shown in the figure indicates an angle of view (vertical angle of view) in the vertical direction of the image in the wide-angle camera 106. Solid lines 309 and 310 indicate two types of vertical angle ranges within the angle of view of the wide-angle camera 106, respectively. The mounting angle of the wide-angle camera 106 is an acute angle with respect to the side surface of the vehicle in the optical axis direction. The angle range indicated by the solid line 309 is a range in which the vicinity of the ceiling or the like is photographed in the case of a station platform, and is often unnecessary for home monitoring purposes. The angle range indicated by the solid line 310 is a range corresponding to the distance from the home, and may be unnecessary depending on the situation. For this reason, it may be more suitable for home monitoring to display an image taken by the wide-angle camera 106 except for the range indicated by the solid line 309 or the solid line 310. By doing in this way, in the limited display area, the area to be visually recognized can be made visible more effectively.

  The area indicated by the symbol L shown in FIG. 15, that is, the area deleted after correction, is assumed to be an area corresponding to the solid line 309 and the solid line 310 in FIG. This is because, as described above, the vicinity of the entrance / exit is the most important in the case of “opening at the entrance / exit”. Thereby, since the vicinity of the entrance / exit is displayed in an enlarged manner, the visibility near the entrance / exit becomes higher. When the operation information is other than “opening at the entrance / exit”, the correction amount may be the same as the correction amount described with reference to FIG. 6 in the first embodiment. As a result, the image processing apparatus 300 can monitor the entire home, and can also visually recognize a run-in from a part away from the entrance / exit.

  Returning to FIG. 14, the processing after the correction amount is determined will be described. The image correction unit 105 acquires the correction amount determined by the correction amount determination unit 303 (step S34). The image correction unit 105 corrects the captured image based on the acquired correction amount (step S35), and then outputs the correction to the image display unit 107 (step S36). As a result, the corrected image is displayed on the image display unit 107.

  As described above, the image processing apparatus 300 according to the present embodiment further includes the correction auxiliary information setting unit 302 that sets the correction auxiliary information used when determining the correction amount. Then, the correction amount determination unit 303 determines the correction amount for the captured image based on the correction auxiliary information (for example, operation information) set by the correction auxiliary information setting unit 302.

  As a result, the image processing apparatus 300 corrects the passenger vehicle 108 such as “running”, “entrance / exit closure”, “entrance / exit opening”, “before / after exit / exit opening / closing”, for example, by correcting based on the operation information. The captured image can be appropriately corrected according to the state. For example, the image processing apparatus 300 can also enlarge and display an area desired to be visually recognized according to the situation by controlling the display range in the vertical direction of the image area of the captured image based on the operation information. Therefore, it is possible to provide an image with high visibility. That is, for example, the image processing apparatus 300 can more effectively visually recognize a region to be visually recognized in a captured image for confirming whether passengers are safely getting on and off the passenger vehicle 108 at home.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, an example in which the image processing apparatuses 100, 200, and 300 according to the first to third embodiments are applied to a vehicle other than a passenger vehicle will be described. The configuration and operation of the image processing apparatus according to this embodiment are the same as those of the image processing apparatuses 100, 200, and 300 described above, and a detailed description thereof is omitted. Although any of the image processing apparatuses 100, 200, and 300 can be applied, the image processing apparatus in the present embodiment will be described as a representative image processing apparatus 100 below.

  FIG. 17 is a diagram for explaining an application example of the image processing apparatus in the present embodiment, and shows an example of application to a store. The illustrated store 400 includes a plurality of entrances 401 and 402 through which persons using the store 400 (hereinafter also referred to as “customers”) can enter and exit. The wide-angle camera 106 is installed above one of the entrances 402. A region surrounded by a broken line 403 and a region surrounded by a broken line 404 in the vicinity of the outside of each of the entrances 401 and 402 indicate attention areas set by the store clerk of the store 400 as a user.

  As described above, for example, in the store 400 having a plurality of entrances and exits, the wide-angle camera 106 is installed so that all the entrances and exits can be photographed as in the case of the passenger vehicle 108, and the image processing apparatuses 100, 200, and 300 are By correcting the photographed image taken by the above, a photographed image with good visibility can be obtained over a wide range.

  In addition, when using correction | amendment auxiliary information like the said 3rd Embodiment, the correction | amendment auxiliary information in the shop 400 is the statistical data regarding the number of customers of the entrance / exit for every time, etc., for example. In this case, for example, the image processing apparatus 300 can perform correction so as to enlarge the attention area near the entrance / exit where there are more users according to the time zone. By doing in this way, the vicinity of the entrance / exit with many users can be visually recognized more clearly.

  In the present embodiment, the installation position of the wide-angle camera 106 is set above the entrance / exit 402. However, for example, by setting it near the main entrance where there are many entrances / exits, the vicinity of the entrance / exit is reflected near the center in the left-right direction of the image. Easy to see. Moreover, if the visibility of all the entrances is taken into consideration, it is desirable to make it near the center of the entrances located at both ends. If it does in this way, it will be easy to ensure the visibility to all the entrances and exits.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above-described embodiment, and includes a design and the like within a scope not departing from the gist of the present invention. For example, the configurations described in the first to fourth embodiments can be arbitrarily combined.

  Note that at least some of the functions of the image processing apparatuses 100, 200, and 300 in the above-described embodiments may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. The “computer system” herein is a computer system built in the image processing apparatuses 100, 200, and 300, and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Further, part or all of the image processing apparatuses 100, 200, and 300 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image processing apparatuses 100, 200, and 300 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

  DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Image input part 102 Camera position setting part 103 Storage part 104,203,303 Correction amount determination part 105 Image correction part 106 Wide angle camera 107 Image display part 201 Object detection part 202 Risk level determination unit, 301 attention area setting unit, 302 correction auxiliary information setting unit

Claims (6)

An image input unit for inputting an image taken by the camera;
In an image in which a reference object on a specific surface is captured, when a predetermined attention area exists in the periphery of the image, an angle on the image of the reference object with respect to the specific surface in the attention area is a predetermined angle. A correction amount determination unit that determines a correction amount for correcting the image so as to be within the range,
An image correction unit that corrects an image input by the image input unit based on the correction amount determined by the correction amount determination unit;
An image processing apparatus comprising: The correction amount determination unit
Further determining the correction amount such that a width of the reference object on the specific surface on the specific area in the region of interest is greater than or equal to a predetermined width;
The image processing apparatus according to claim 1. A risk determination unit that determines a risk based on the input image;
The correction amount determination unit
Determining the correction amount according to the risk determined by the risk determination unit;
The image processing apparatus according to claim 1. A correction auxiliary information setting unit for setting correction auxiliary information used when determining the correction amount;
The correction amount determination unit
Determining the correction amount based on the correction auxiliary information set by the correction auxiliary information setting unit;
The image processing apparatus according to claim 1. The correction amount determination unit
When determining the correction amount, the correction amount is determined so as to enlarge an area excluding a part of the image area of the image;
The image processing apparatus according to any one of claims 1 to 4. An image input step for inputting an image taken by the camera;
In an image in which a reference object on a specific surface is captured, when a predetermined attention area exists in the periphery of the image, an angle on the image of the reference object with respect to the specific surface in the attention area is a predetermined angle. A correction amount determining step for determining a correction amount for correcting the image so as to be within the range,
An image correction step for correcting the image input by the image input step based on the correction amount determined by the correction amount determination step;
An image processing method including:

Images