JP2020074186A - Image processing device, image processing system, image processing method, and image processing program - Google Patents

Abstract

To provide an image processing device that enables a user to clearly and visually recognize images without impairing the continuity thereof.SOLUTION: An image processing device includes: an image capturing unit configured to output a plurality of images captured in ranges aligned in the horizontal direction with respect to an apparatus mounted thereon; an image composition unit configured to compose the plurality of images output from the image capturing unit so as to be arranged in a lateral direction X to generate a composite image; a boundary line drawing unit configured to draw a boundary line KL so as to be overlapped with the boundary portion between neighboring images; and an image display unit 12 configured to display the composite image with the drawn boundary line in a display area HR. The boundary line drawing unit variably draws the width of the boundary line KL based on a drawing ratio preset with respect to the width of the adjacent image in the lateral direction X.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and an image processing program that perform processing on an image captured around a mounted device.

  2. Description of the Related Art In recent years, mobile robots that autonomously travel on a traveling surface have been mounted with a camera that captures an image of the surroundings, and have been used for the purpose of monitoring a wide area on behalf of a user. By the way, depending on the performance of the camera and the like, there are cases in which the range that can be photographed is restricted and it is not possible to fit a wide range in one image. On the other hand, a method is used in which a plurality of images with different shooting ranges are combined to form an integrated image showing a wide range (for example, refer to Patent Document 1).

JP-A-5-183789

  The electronic still camera described in Patent Document 1 divides an object into a plurality of areas to obtain a combined image with a large angle of view, and includes a storage unit that stores the image, a combining unit that combines the images, and a plurality of combining units. Display means for displaying the imaging situation of the area. In the electronic still camera described above, the boundary portions of adjacent images are likely to be similar images, so that there is a problem that the switching portion of the images is difficult to understand and the user is confused. Further, although adding a boundary line to a boundary portion of an image has been studied, if the boundary line is too thick, there is a problem that the continuity of the image is impaired and the angle of view of the image is limited.

  The present invention has been made to solve the above problems, and by drawing a boundary line having an appropriate thickness with respect to the width of an image, the user can be prevented from losing the continuity of the image. An object of the present invention is to provide an image processing device, an image processing system, an image processing method, and an image processing program that can be clearly recognized.

  An image processing apparatus according to the present invention includes an image capturing unit that outputs a plurality of images captured in a range aligned in a horizontal direction with respect to a mounted device, and the plurality of images output from the image capturing unit in a horizontal direction. Between the adjacent images of the plurality of images that are the source of the combined image in the combined image generated by the image combining unit An image processing apparatus comprising: a boundary line drawing unit that draws a boundary line overlapping a boundary part of the image; and an image display unit that displays the combined image in which the boundary line is drawn in a display area. Among the plurality of images that are the source of the image, the adjacent image is an adjacent image, and the boundary line drawing unit sets the width of the boundary line to a drawing ratio set in advance with respect to the width of the adjacent image in the horizontal direction. Based on And wherein the Rukoto.

  An image processing apparatus according to the present invention includes an image capturing unit that outputs a plurality of images captured in a range aligned in a horizontal direction with respect to a mounted device, and the plurality of images output from the image capturing unit in a horizontal direction. And an image combining unit that generates a combined image by combining them so that they are arranged side by side in the vertical direction, and in the combined image generated by the image combining unit, adjacent images among the plurality of images that are the source of the combined image An image processing apparatus comprising: a boundary line drawing unit that draws a boundary line so as to overlap a boundary part between them; and an image display unit that displays the composite image in which the boundary line is drawn in a display area, Of the plurality of images that are the source of the composite image, adjacent images are adjacent images, and the boundary line drawing unit sets the width of the boundary line to the width in the horizontal direction or the width in the vertical direction of the adjacent image. Preset for either Based on the drawing ratio, characterized by varying to draw.

  In the image processing device according to the present invention, the image display unit displays the plurality of images output from the image capturing unit side by side in the display area in the horizontal direction and the vertical direction, and the boundary line drawing unit is configured to display the image. The widths of the boundary lines in the horizontal and vertical directions may be set based on either the horizontal width or the vertical width of the adjacent images.

  In the image processing device according to the present invention, a brightness detection unit that detects the brightness of the image output from the image capturing unit is provided, and the boundary drawing unit, when the brightness of the image is higher than a preset reference value, The boundary line may be drawn in a dark color, and when the brightness of the image is lower than a preset reference value, the boundary line may be drawn in a light color.

  In the image processing device according to the present invention, the boundary line drawing unit may be configured to change the drawing ratio according to a user's instruction.

  An image processing system according to the present invention includes the image processing device according to the present invention, wherein the device is a moving body traveling on a traveling surface, and transmits an image captured through communication to the image display unit. Characterize.

  An image processing method according to the present invention is an image processing method in an image processing apparatus including an image shooting unit that outputs a plurality of images obtained by shooting a range aligned in a horizontal direction with respect to a mounted device. An image synthesizing step of synthesizing the plurality of images output from the unit so as to be arranged in a horizontal direction to generate a synthetic image; and the synthetic image generated in the image synthesizing step, which is a source of the synthetic image. Of a plurality of images, a boundary line drawing step of drawing a boundary line to overlap a boundary portion between adjacent images, and an image display step of displaying the composite image in which the boundary line is drawn in a display area. Of the plurality of images that are the source of the composite image, adjacent images are adjacent images, and the boundary line drawing step determines the width of the boundary line with respect to the lateral width of the adjacent image. Based on a preset drawing ratio, characterized by varying to draw.

  An image processing program according to the present invention is characterized by causing a computer to execute each step of the image processing method according to the present invention.

  According to the present invention, by drawing a boundary line having an appropriate thickness with respect to the width of an image, it is possible for the user to clearly see the boundary between images. In addition, by adjusting the width of the boundary line, it is possible to maintain the continuity of the adjacent images, secure a large area for arranging the images, and increase the size of the images to improve the visibility.

It is a schematic top view of the mobile body which images the surroundings. It is a schematic structure figure showing a schematic structure of an image processing device concerning a 1st embodiment of the present invention. It is a schematic structure figure showing a schematic structure of a surveillance system using an image processing device concerning a 1st embodiment of the present invention. It is explanatory drawing which shows the 1st display image displayed on a display area. It is a flow figure showing a processing flow of an image processing method in an image processing device concerning a 1st embodiment of the present invention. It is explanatory drawing which shows the 2nd display image displayed on a display area. It is a flow figure showing the processing flow of the image processing method in the image processing device concerning a 3rd embodiment of the present invention.

  An image processing apparatus according to the first embodiment of the present invention will be described below with reference to the drawings.

  FIG. 1 is a schematic top view of a moving body that captures an image of the surroundings.

  The moving body 1 shown in FIG. 1 is a four-wheel vehicle that moves on a traveling surface along a preset route. The moving body 1 normally goes straight ahead (upward in FIG. 1) and changes the traveling direction as appropriate. Note that, for simplicity of description, the direction in which the moving body 1 goes straight ahead is referred to as a straight running direction S, the front in the straight running direction S is omitted as the front, and the rear in the moving direction of the moving body 1 (see FIG. In 1, the lower part may be omitted as the rear part. Of the directions orthogonal to the straight traveling direction S, one may be referred to as left (left in FIG. 1) and the other may be referred to as right (right in FIG. 1).

  The mobile body 1 is a mounted camera for capturing images of the surrounding area. The front camera 2a captures the front of the mobile body 1, the right camera 2b captures the right side of the mobile body 1, and the rear side of the mobile body 1. The rear camera 2c and the left camera 2d that captures the left side of the moving body 1 are provided. The four on-board cameras are configured to capture images and videos using a wide-angle lens, and capture a range that expands by about 190 ° in a top view with respect to the direction in which the on-board cameras are oriented. Specifically, the front camera 2a outputs the front range SR1 in front of the moving body 1 as a front image GZ (see FIG. 4 described later). Similarly to the front camera 2a, the right camera 2b outputs the right range SR2 on the right side of the moving body 1 as a right image GU, and the rear camera 2c displays the rear range SR3 behind the moving body 1 on the rear image GK. The left camera 2d outputs the left range SR4 on the left side of the moving body 1 as a left image GS. The four images output from the onboard camera are processed by the image processing device 10 described later so as to be displayed simultaneously. The image captured by the onboard camera is not limited to a still image and may be a moving image. Further, hereinafter, in order to distinguish it from other images, the images captured by the image capturing unit 11 such as the front image GZ, the rear image GK, the left image GS, and the right image GU may be referred to as captured images.

  FIG. 2 is a schematic configuration diagram showing a schematic configuration of the image processing apparatus according to the first embodiment of the present invention.

  The image processing apparatus 10 according to the first embodiment of the present invention includes an image capturing unit 11, an image display unit 12, a boundary line drawing unit 13, an image synthesizing unit 14, an instruction input unit 15, and a brightness detection unit 16. Has been done.

  The image capturing unit 11 captures an image of the surroundings of the mounted device (particularly, the moving body 1). In the present embodiment, the mounted cameras (front camera 2a, rear camera 2c, left camera 2d, and right camera 2b) correspond to the image capturing unit 11. The image capturing unit 11 is not limited to the configuration shown in FIG. 1, and the number and arrangement of the mounted cameras may be changed.

  The image display unit 12 is, for example, a display, and displays the image captured by the image capturing unit 11 in the display area HR. The boundary line drawing unit 13 draws a boundary line KL (see FIG. 4 described later) on the image displayed on the image display unit 12. The image synthesizing unit 14 processes the image output from the image capturing unit 11 to synthesize a plurality of images and integrate them. The instruction input unit 15 is configured to include, for example, keys and buttons, receives an instruction from the user, and sets the style of the image displayed in the display area HR. It should be noted that a specific example and style of the display image displayed by the image display unit 12 will be described in detail with reference to FIG. 4 described later.

  The brightness detection unit 16 is related to the third embodiment, and detailed description will be given later. That is, in the first embodiment, whether or not the brightness detection unit 16 is provided may be changed as appropriate.

  FIG. 3 is a schematic configuration diagram showing a schematic configuration of a monitoring system using the image processing apparatus according to the first embodiment of the present invention.

  The image processing device 10 described above is applied to a monitoring system that monitors the surroundings by using a plurality of moving bodies 1. Specifically, the monitoring system (image processing system) includes a monitoring device 20 and a plurality of mobile bodies 1 connected via a network NW. That is, the image captured by the moving body 1 is transmitted to the monitoring device 20 and displayed on the display of the monitoring device 20.

  The present invention is not limited to this, and may be configured by only one of the moving body 1 and the monitoring device 20, and the monitoring device 20 may be provided with the image capturing unit 11. In the case where the mobile body 1 alone is used, the mobile body 1 may perform image processing on the captured image, transmit the image to the outside, or store the image. The boundary line drawing unit 13 and the image synthesizing unit 14 described above are stored as a program installed in advance in a CPU (not shown), and execute the stored program to execute the process described below. The CPU that stores the boundary line drawing unit 13 and the image synthesizing unit 14 may be provided in either or both of the moving body 1 and the monitoring device 20 depending on the configuration of the image processing system.

  FIG. 4 is an explanatory diagram showing a first display image displayed in the display area.

  The display area HR has a rectangular shape in which the horizontal direction X is wider than the vertical direction Y. In the present embodiment, the captured images are arranged side by side in the display area HR, and the adjacent images are referred to as adjacent images, and the direction in which the adjacent images are arranged is called the adjacent direction. In the first display image HG1, four captured images of a front image GZ, a rear image GK, a left image GS, and a right image GU are arranged. In the first display image HG1, four images are arranged in a row in the horizontal direction X. That is, in the present embodiment, the adjacent direction is the horizontal direction X. The rear image GK is divided into two, that is, a first rear divided image GK1 and a second rear divided image GK2. Specifically, the four images are arranged from the left of the display region HR in the order of the first rear divided image GK1, the left image GS, the front image GZ, the right image GU, and the second rear divided image GK2. The front image GZ is arranged in the center of the display area HR. In other words, by arranging the images of the front, back, left, and right of the device side by side in the horizontal direction X, it is possible to display the images as a panoramic integrated image.

  Note that the captured image captured by the image capturing unit 11 may be subjected to processing such as correction in advance. In other words, in a wide-angle camera that captures a wide range, distortion such as bending of the outer edge portion of the image may occur, so distortion correction is performed to eliminate this. Further, cropping processing for changing the size according to the position where the image is arranged may be performed. Further, regarding a portion where adjacent images overlap with each other, it is only necessary to grasp the photographed range from the coordinates set for the position of the device and delete the extra area. By appropriately performing these processes, it is possible to perform a combining process that integrates a plurality of images.

  By the way, since a plurality of captured images have different shooting directions, when the images are combined, the images may become discontinuous, such as the images not being aligned at the boundary portion and being displaced. This becomes particularly clear when the image is a moving image, for example, when an object in the image moves, which may give the user a feeling of strangeness.

  Therefore, in the present embodiment, the boundary line KL is drawn between adjacent images (for example, between the left image GS and the front image GZ). In this way, when the boundary line KL is drawn at the boundary portion, discontinuous portions of adjacent images are hidden, but according to human visual recognition, if the images are related to each other, the hidden portion is complemented for recognition. Is known to do. By utilizing this, the user can recognize that the adjacent images are continuous by drawing the boundary line KL.

  The width (boundary line width LW) of the boundary line KL in the adjacent direction (horizontal direction X) is set in advance with respect to the adjacent direction width (image width PW) of the adjacent image (for example, the left image GS). It is drawn based on the ratio. In the present embodiment, the drawing ratio of the boundary line width LW to the image width PW is set to 1%, but the present invention is not limited to this, and the drawing ratio is set to 0.5% or more and 1.5% or less. Just do it. Therefore, by drawing the boundary line KL having an appropriate thickness with respect to the image width PW, the user can clearly see the boundary between the images. Further, by adjusting the boundary line width LW, it is possible to maintain the continuity of the adjacent images, secure a wide area for arranging the images, and increase the size of the images to improve the visibility. When the boundary line KL is thin, the boundary line KL is no longer recognized, and the effect of alleviating the discomfort of the boundary portion is weakened. When the boundary line KL is thick, the range of the boundary line KL missing is wide. Therefore, there is a concern that the image becomes smaller. Considering these things, it is desirable that the boundary line KL be set to have an appropriate thickness.

  The boundary line KL may be drawn so as to overlap the boundary portion of the combined image. The calculation method of the image width PW may be appropriately set according to the drawing method of the boundary line KL. The color of the boundary line KL may be set in advance, or may be specified by the user via the instruction input unit 15. Furthermore, the boundary line width LW may be changed as appropriate as long as it is within the range set as the drawing ratio. For example, a plurality of settings regarding the boundary line width LW may be provided, and the boundary line KL may be made thin or thick according to a user's instruction, or the boundary line KL may be hidden.

  Next, a processing flow of the image processing method in the image processing apparatus 10 will be described with reference to the drawings.

  FIG. 5 is a flowchart showing the processing flow of the image processing method in the image processing apparatus according to the first embodiment of the present invention.

  In step S01, the image composition unit 14 acquires the width of the captured image (image width PW). The image width PW may be appropriately adjusted according to the size of the display area HR and the number of images to be displayed.

  In step S02, the boundary line drawing unit 13 calculates the boundary line width LW.

  In step S03, the image combining unit 14 combines the captured images.

  In step S04, the boundary line drawing unit 13 draws the boundary line KL, and the process ends.

  An image processing method according to the present invention is an image processing method in an image processing apparatus 10 including an image capturing unit 11 that captures an image of the surroundings of a device (for example, a moving body 1) mounted on the image display unit 12. In the image display step of displaying the image captured by the image capturing unit 11 in the display region HR, and in the boundary line rendering unit 13, a boundary line KL is drawn between the images arranged adjacent to each other in the display region HR. And a boundary line drawing step that causes adjacent images to be arranged in the display area HR, and a direction in which the adjacent images are arranged is an adjacent direction. The boundary line width LW set as the width in the adjacent direction is drawn based on a drawing ratio set in advance for the width in the adjacent direction of the adjacent images (image width PW).

(Second embodiment)
Next, an image processing apparatus according to the second embodiment of the present invention will be described with reference to the drawings.

  FIG. 6 is an explanatory diagram showing a second display image displayed in the display area.

  In the second embodiment, the content displayed on the image display unit 12 is different from that of the first embodiment. The second embodiment has substantially the same configuration as that of the first embodiment shown in FIGS. 1 to 3, and therefore the description thereof will be omitted and only different points will be described. In the first embodiment, the areas to be photographed by the image photographing unit 11 are arranged in the horizontal direction, but in the second embodiment, they are arranged in the vertical direction in addition to the horizontal direction. That is, the second embodiment is different from the first embodiment in the range to be photographed by the camera. For example, two cameras for capturing the front range SR1 may be provided, and each of the cameras may capture a different range in the vertical direction. Further, when the number of cameras is one, the shooting range may be changed by appropriately changing the direction of the cameras.

  In the second display image HG2, the upper left image GLU, the upper right image GRU, the lower left image GLD, and the lower right image GRD are arranged in the vertical direction Y and the horizontal direction X. The upper left image GLU, the upper right image GRU, the lower left image GLD, and the lower right image GRD are output from the image capturing unit 11 and are captured images in a range aligned in the horizontal direction or the vertical direction. Specifically, the upper left image GLU is arranged at the upper left of the display area HR, the upper right image GRU is arranged at the upper right of the display area HR, the lower left image GLD is arranged at the lower left of the display area HR, and the lower right of the display area HR. The lower right image GRD is arranged at. That is, the upper left image GLU is aligned with the upper right image GRU in the horizontal direction X, and is aligned with the lower left image GLD in the vertical direction Y. The lower right image GRD is aligned with the lower left image GLD in the horizontal direction X, and is aligned with the upper right image GRU in the vertical direction Y. All four images have the same size. In the second display image HG2, all four images have the same size, and the four images are evenly allocated to the display area HR.

  A horizontal boundary line KLX is drawn between the upper left image GLU and the upper right image GRU and between the lower left image GLD and the lower right image GRD, and between the upper left image GLU and the lower left image GLD and between the upper right image GRU and the lower right image. A vertical boundary line KLY is drawn between the images GRD.

  As for the horizontal boundary line KLX and the vertical boundary line KLY, the boundary line width LW is set based on the image width PW of the captured image in the horizontal direction X. In other words, the horizontal direction X and the vertical direction Y are border lines having the same thickness, and the images are arranged at a constant interval. Therefore, it is possible to unify the image arrangement and display without giving a feeling of strangeness to the user. .. The boundary line width LW may be set based on the image width PW in the vertical direction Y of the captured image.

(Third Embodiment)
Next, an image processing apparatus according to the third embodiment of the present invention will be described with reference to the drawings.

  FIG. 7 is a flowchart showing the processing flow of the image processing method in the image processing apparatus according to the third embodiment of the present invention.

  In the third embodiment, the processing flow of the image processing method is different from that of the first embodiment. Since the third embodiment has almost the same configuration as the first embodiment shown in FIGS. 1 to 3, the description thereof will be omitted and only different points will be described.

  The third embodiment is configured to set the color of the boundary line KL according to the captured image. Specifically, the brightness detection unit 16 detects the brightness of the image output from the image capturing unit 11. The brightness detected by the brightness detection unit 16 is output to the boundary line drawing unit 13. The boundary line drawing unit 13 draws the boundary line KL in a dark color (for example, black) when the brightness of the image is higher than a preset reference value, and when the brightness of the image is lower than the preset reference value, the boundary line KL The line KL is drawn in a bright color (for example, white). Therefore, by setting the color of the boundary line KL so that the brightness is opposite to that of the image, it is possible to make the boundary line KL conspicuous and make the boundary of the image easier to understand. Note that the color of the boundary line KL is not limited to black and white, and may be another color such as red or blue, and it is possible to switch between a dark color and a light color by setting different lightness.

  Since step S11, step S12, step S13, and step S15 shown in FIG. 7 are the same as step S01 to step S04 shown in FIG. 5, description thereof will be omitted. That is, in the third embodiment, step S14 is performed between step S13 and step S15.

  In step S14, the brightness detection unit 16 detects the brightness of the captured image. Then, in step S15, the color of the boundary line KL is set based on the result of step S14.

  The embodiment disclosed this time is an example in all respects, and is not a basis for a limited interpretation. Therefore, the technical scope of the present invention should not be construed only by the above-described embodiments, but should be defined based on the description of the claims. Also, the meaning equivalent to the scope of the claims and all modifications within the scope are included.

DESCRIPTION OF SYMBOLS 1 Moving object 10 Image processing device 11 Image capturing unit 12 Image display unit 13 Boundary line drawing unit 14 Image combining unit 15 Instruction input unit 16 Brightness detection unit 20 Monitoring device HR display area KL Borderline LW Borderline width PW Image width X Horizontal Direction Y Vertical

Claims (8)

An image capturing unit that outputs a plurality of images captured in a range aligned in the horizontal direction with respect to the mounted device,
An image combining unit that generates a combined image by combining the plurality of images output from the image capturing unit so as to be arranged in the horizontal direction,
In the composite image generated by the image composition unit, a boundary line drawing unit that draws a boundary line overlapping a boundary portion between adjacent images among the plurality of images that are the source of the composite image,
An image processing apparatus comprising: an image display unit that displays the composite image in which the boundary line is drawn in a display area,
Of the plurality of images that are the source of the composite image, adjacent images are adjacent images,
The image processing apparatus, wherein the boundary line drawing unit changes and draws the width of the boundary line based on a drawing ratio set in advance with respect to the width of the adjacent image in the horizontal direction. An image capturing unit that outputs a plurality of images captured in a range aligned in the horizontal direction with respect to the mounted device,
An image combining unit that generates a combined image by combining the plurality of images output from the image capturing unit so as to be aligned in the horizontal direction and the vertical direction,
In the composite image generated by the image composition unit, a boundary line drawing unit that draws a boundary line overlapping a boundary portion between adjacent images among the plurality of images that are the source of the composite image,
An image processing apparatus comprising: an image display unit that displays the composite image in which the boundary line is drawn in a display area,
Of the plurality of images that are the source of the composite image, adjacent images are adjacent images,
The boundary line drawing unit may change the width of the boundary line based on a drawing ratio set in advance for either the width in the horizontal direction or the width in the vertical direction of the adjacent image, and draw the variable width. A characteristic image processing device. The image processing apparatus according to claim 2, wherein
The image processing apparatus, wherein the border line drawing unit sets the border lines in the horizontal direction and the vertical direction to the same width. The image processing apparatus according to any one of claims 1 to 3,
A brightness detection unit that detects the brightness of the image output from the image capturing unit,
The border line drawing unit draws the border line in a dark color when the lightness of the image is higher than a preset reference value, and draws the border line in a light color when the lightness of the image is lower than the preset reference value. Image processing device characterized by drawing with. The image processing apparatus according to any one of claims 1 to 4,
The image processing apparatus, wherein the boundary line drawing unit changes the drawing ratio according to a user's instruction. An image processing system comprising the image processing device according to claim 1.
The image processing system, wherein the device is a moving body that travels on a traveling surface, and transmits the captured image to the image display unit via communication. An image processing method in an image processing apparatus, comprising: an image capturing section that outputs a plurality of images captured in a range aligned in a horizontal direction with respect to a mounted device
An image combining step of generating a combined image by combining the plurality of images output from the image capturing unit so as to be arranged in the horizontal direction,
In the composite image generated in the image combining step, a boundary line drawing step of drawing a boundary line to overlap a boundary portion between adjacent images among the plurality of images serving as the source of the composite image,
An image display step of displaying the composite image in which the boundary line is drawn in a display area,
Of the plurality of images that are the source of the composite image, adjacent images are adjacent images,
The image processing method, wherein in the boundary line drawing step, the width of the boundary line is changed and drawn based on a drawing ratio preset with respect to a width of the adjacent image in the horizontal direction.   An image processing program for causing a computer to execute the steps according to claim 7.

Images