JP6784199B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing device that processes an image taken in the hoistway of an elevator.

In elevators, cameras are installed in cages to take pictures of the inside of the hoistway for inspection and abnormality detection. At that time, since it is inconvenient to take pictures of all four sides of the hoistway separately in taking pictures using a normal camera, the use of a spherical camera is being considered. The omnidirectional camera is a camera that performs 360-degree panoramic photography in all directions, up, down, left, and right, and is also called an omnidirectional camera.

The image taken by the omnidirectional camera can be divided into a plurality of areas by performing image processing, and the divided areas can be displayed on the screen. For example, Patent Document 1 describes the division in this way.

Japanese Unexamined Patent Publication No. 2016-54423 (published on April 14, 2016)

In the hoistway of the elevator, still images are often taken at a plurality of positions along the moving direction of the car, or moving images are taken while moving along the moving direction of the car. When an omnidirectional camera is used for this shooting, it is a complicated task to individually perform the above-mentioned region division for each frame of the still image or the moving image shot at each position.

On the other hand, in the technique disclosed in Patent Document 1, an image taken by an omnidirectional camera is divided into a specific number of image regions in the camera. Using such a spherical camera eliminates the complexity of dividing the area, but since the boundary of each image area has been specified by the camera, an important point in inspection and abnormality detection is the boundary. It may overlap, causing inconvenience in inspection and abnormality detection.

In order to avoid this problem, it is conceivable to appropriately adjust the shooting direction of the spherical camera at the time of shooting, but such adjustment work causes complexity at the time of shooting.

In view of the above problems, an object of the present invention is to provide an image processing apparatus capable of suppressing the above complexity.

In order to solve the above problems, the image processing device according to one aspect of the present invention has a break in the 360-degree image when the 360-degree image taken by the 360-degree camera is expanded and displayed on a flat image. A boundary information acquisition unit that acquires boundary information that defines the boundary, and a development unit that expands each 360-degree image into a flat image by applying the boundary defined by the boundary information to a plurality of 360-degree images. And.

According to the above configuration, each of the plurality of 360-degree images can be developed into a plane image based on the acquired boundary information. Therefore, it is possible to suppress the complexity of individually designating boundaries for each of the plurality of 360-degree images. In addition, if the boundary is set so as not to overlap with the important points in inspection and abnormality detection, it is not necessary to adjust the shooting direction of the 360-degree camera when shooting a 360-degree image, and therefore it is complicated to adjust the shooting direction. Can be suppressed.

The boundary information may be any information that specifies the position of the image data in the format, such as coordinates in the image data constituting the 360-degree image.

Further, the above-mentioned "360 degree image" includes an "all-around image" corresponding to projection on a cylindrical screen, an "omnidirectional image" corresponding to projection on a hemispherical screen, and "all-around image" corresponding to projection on a global screen. "Spherical image" is included.

Further, the image may be a still image or one frame of a series of moving images. Further, the image processing device may be built in the camera, but may be realized as, for example, a personal computer separately from the camera.

Further, in the image processing apparatus according to one aspect of the present invention, a portion satisfying a predetermined condition is specified by image recognition processing for the image data of the 360-degree image, and boundary information with that portion as the boundary is generated. A first boundary information generation unit that is provided to the boundary information acquisition unit may be further provided.

A 360-degree image taken by a 360-degree camera inside the elevator hoistway is suitable as a boundary when a 360-degree image is expanded and displayed on a flat image, such as the four corners of the hoistway and the four corners of a car, and image recognition. The characteristic part that can be identified by the processing is shown.

According to the above configuration, the feature portion can be specified by the boundary information generation unit, and the boundary information having the feature portion as the boundary can be generated. As a result, it is possible to automatically set an appropriate boundary and develop a flat image.

Further, in the image processing device according to one aspect of the present invention, the position of the boundary is specified to the user based on the image display unit for displaying the 360-degree image on the display device and the 360-degree image displayed by the image display unit. A second boundary information generation unit, which generates boundary information corresponding to the position and provides the boundary information to the boundary information acquisition unit, may be further provided.

According to the above configuration, the user needs to specify the position of the boundary while looking at the displayed 360-degree image, but the position of the specified boundary with respect to one 360-degree image is set to the other 360 in the developing unit. Since it can be applied to the image, it is possible to avoid complicated work required by the user.

Further, in the image processing apparatus according to one aspect of the present invention, the boundary information acquisition unit divides and develops a 360-degree image into four plane images corresponding to four different orientations located on a plane. You may get the necessary boundary information.

Elevator hoistways are usually approximately rectangular parallelepiped. Therefore, as described in the above configuration, the boundary information acquisition unit acquires the boundary information necessary for developing the 360-degree image while dividing the 360-degree image into four plane images corresponding to four different directions located on the plane. By developing a 360-degree image into a flat image based on this boundary information, it becomes possible to divide the image into a region in which each side wall in the hoistway is the front surface. This facilitates inspection and abnormality detection based on the divided images.

Further, the image processing apparatus according to one aspect of the present invention may further include a screen generation unit that generates a screen in which four plane images developed by the development unit are arranged side by side.

According to the above configuration, since the images of each side wall in the hoistway can be displayed in a horizontally expanded state, inspection and abnormality detection based on the divided images become easier.

Further, in the image processing apparatus according to one aspect of the present invention, the screen generation unit may generate a screen in which a set of horizontally arranged plane images corresponding to each 360-degree image is vertically arranged.

According to the above configuration, by targeting a plurality of 360-degree images taken along the vertical direction in the hoistway, it is possible to list a plurality of shooting positions vertically expanded for each side wall. Therefore, inspection and abnormality detection based on the divided images become easier.

According to one aspect of the present invention, it is complicated to individually specify a boundary that becomes a break of a 360-degree image for each of a plurality of 360-degree images when the 360-degree image is expanded and displayed on a flat image. It has the effect of being able to be suppressed.

In addition, if the boundary is set so as not to overlap with the important points in inspection and abnormality detection, it is not necessary to adjust the shooting direction of the 360-degree camera when shooting a 360-degree image, and therefore it is complicated to adjust the shooting direction. Can be suppressed.

It is a block diagram which shows an example of the main part structure of the image processing apparatus included in the image processing system which concerns on Embodiment 1 of this invention. It is the schematic which shows an example of the structure of the elevator shown in FIG. It is a figure which shows an example of the spherical image displayed by the display device shown in FIG. It is a figure which shows an example of the plane image displayed by the display device shown in FIG. It is a flowchart which shows an example of the flow of the image development processing executed by the image processing apparatus shown in FIG. It is a block diagram which shows an example of the main part structure of the image processing apparatus included in the image processing system which concerns on Embodiment 2 of this invention. It is a flowchart which shows an example of the flow of the image development processing executed by the image processing apparatus shown in FIG. It is a block diagram which shows an example of the main part structure of the image processing apparatus included in the image processing system which concerns on Embodiment 3 of this invention. (A) is a figure which shows the boundary for generating a divided image. FIG. 8B is a diagram showing an example of a divided image displayed by the display device shown in FIG. It is a flowchart which shows an example of the flow of the image development processing executed by the image processing apparatus shown in FIG. It is the schematic which shows the other structural example of an elevator. It is the schematic which shows the further other structural example of an elevator.

[Embodiment 1]
Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 5.

(Overview of elevator 20)
First, the basic structure of the elevator 20 according to the present embodiment will be described with reference to FIG. FIG. 2 is a schematic view showing an example of the configuration of the elevator 20. The schematic configuration of the elevator 20 is not limited to the illustrated example.

The illustrated elevator 20 is an elevator provided with a hoistway 21 and a machine room 23. The position of the machine room 23 is not limited to the upper part of the hoistway 21. Further, the elevator 20 may be configured not to include the machine room 23.

As shown, the car 22 of the elevator 20 is connected to one end of the rope. A balancing weight is connected to the other end of the rope. The hoisting machine installed in the machine room 23 winds up the rope, so that the car 22 moves up and down in the hoistway 21.

In the machine room 23, a control panel 24 that controls each part of the elevator 20 is installed. The control panel 24 moves the car 22 to the floor designated by the user, stops on the floor, and stops the car in response to control signals acquired from various buttons (for example, a landing button) provided on the elevator 20. 22 and the opening and closing of the door provided in the landing, the display change of the position display provided in the landing, and the like are controlled.

Further, the control panel 24 may be configured to be able to communicate with an external device via a predetermined communication network (for example, the Internet), and the image processing device according to the present embodiment may be configured to communicate with the external device via the communication network. You may communicate with 1.

(Spherical camera 2)
As shown in the figure, the spherical camera 2 is installed in the car 22 according to the present embodiment. The omnidirectional camera 2 captures a still image in the hoistway 21 at a plurality of positions along the moving direction of the car 22 and a moving image in the hoistway 21 while moving along the moving direction of the car 22. Do at least one of the above to generate an image. The omnidirectional camera 2 is a so-called "360-degree camera" that performs 360-degree panoramic photography in all directions, up, down, left, and right. In other words, the image generated by the spherical camera 2 is a so-called "spherical image" (360-degree image) corresponding to the projection on the spherical screen. Further, the omnidirectional camera 2 according to the present embodiment is a camera that automatically takes a picture by remote control of a worker or at predetermined intervals.

In the present embodiment, the spherical camera 2 will be described as an example of the camera installed in the car 22, but the present invention is not limited to this. For example, the camera installed in the car 22 may be a camera that generates a so-called all-around image corresponding to projection on a cylindrical screen, or may generate a so-called all-sky image corresponding to projection on a hemispherical screen. It may be a camera that does.

In the illustrated example, the spherical camera 2 is installed on the outer upper part of the car 22. However, the installation position of the spherical camera 2 is not limited to the illustrated example. The omnidirectional camera 2 may be installed at a position where the inside of the hoistway 21 can be photographed, and may be installed, for example, at the lower outer side of the car 22.

Further, the spherical camera 2 does not have to be always installed in the car 22. Specifically, the spherical camera 2 may be brought into the car 22 by a worker who inspects the elevator 20 during maintenance and inspection. Further, the spherical camera 2 may be used in a state where it is not installed in the car 22. Specifically, the worker on the car 22 may take a picture of the inside of the hoistway 21 while holding the spherical camera 2 in his hand. In the case of this example, the spherical camera 2 does not have to have a function of taking a picture by remote control of an operator or a function of taking a picture automatically at predetermined intervals.

(Outline of image processing system 100)
Next, an outline of the image processing system 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing an example of a main configuration of an image processing device 1 included in the image processing system 100. As shown in the figure, the image processing system 100 includes an image processing device 1, the above-mentioned spherical camera 2, a storage device 3, an input device 4, and a display device 5. The image processing device 1 is a so-called computer that processes and outputs an input image.

The input device 4 accepts a user's operation and inputs a signal (illustrated operation signal) corresponding to the operation to the image processing device 1. The input device 4 is, for example, a mouse and a keyboard. The display device 5 acquires and displays an image (display data shown in the figure) generated by the image processing device 1. The display device 5 is, for example, a liquid crystal display. Further, the input device 4 and the display device 5 may be integrated with the image processing device 1.

The storage device 3 stores the omnidirectional image generated by the omnidirectional camera 2. The storage device 3 is typically a storage device that can be physically connected to the image processing device 1 such as an SD (Secure Digital) memory card that can be attached to and detached from the spherical camera 2. It may be a storage device capable of communicating with the spherical camera 2 and the image processing device 1 via the communication network (for example, the Internet). When the spherical camera 2 can communicate with the image processing device 1 by wire or wirelessly, and the spherical image generated by the spherical camera 2 is directly transmitted to the image processing device 1. , The image processing system 100 does not have to include the storage device 3.

(Main part configuration of image processing device 1)
Next, the configuration of the main part of the image processing apparatus 1 will be described with reference to FIG. As shown in the figure, the image processing device 1 includes a control unit 10, a storage unit 11, and a communication unit 12. The control unit 10 controls each unit of the image processing device 1 in an integrated manner. The details of the control unit 10 will be described later.

The storage unit 11 stores various data used by the image processing device 1. The storage unit 11 according to the present embodiment stores the spherical image generated by the spherical camera 2 and the planar image generated by developing the spherical image by the image processing device 1. A flat image is an image that corresponds to the projection of a flat screen onto a screen. In addition, the storage unit 11 stores an application for displaying the spherical image on the display device 5 (hereinafter, referred to as “spherical image viewer”). The storage unit 11 may be a built-in memory of the image processing device 1 or a hard disk externally attached to the image processing device 1.

The communication unit 12 connects to an external device in a wired or wireless manner, and acquires data from the connected external device. The communication unit 12 according to the present embodiment acquires a spherical image from the storage device 3 and stores the acquired spherical image in the storage unit 11. The communication unit 12 may acquire a spherical image from the storage device 3 by using an operation signal from the input device 4 as a trigger. Further, the communication unit 12 may automatically acquire a spherical image from the storage device 3 when the image processing device 1 and the storage device 3 are communicably connected to each other.

As shown in the figure, the control unit 10 includes an input control unit 101, a boundary information generation unit 102 (second boundary information generation unit), a development unit 103, and a display control unit 105 (image display unit).

The input control unit 101 acquires an operation signal from the input device 4 and outputs an instruction corresponding to the operation signal to each unit of the control unit 10. In particular, when the input control unit 101 acquires an operation signal indicating an operation for displaying a spherical image or a plane image on the display device 5, an instruction for displaying these images on the display device 5 (illustrated image display instruction) is given. Output to the display control unit 105. Further, the input control unit 101 is an operation of designating a position serving as a starting point of a boundary which is a break of the spherical image when the spherical image is expanded and displayed on a flat image (hereinafter, "designating operation"). When the operation signal indicating (referred to as) is acquired, an instruction for generating boundary information defining the boundary (hereinafter referred to as “generation instruction”) is output to the boundary information generation unit 102. In the generation instruction, information indicating a designated position (hereinafter referred to as "position information") and information indicating an omnidirectional image on which the designated operation has been performed (hereinafter referred to as "image information"). )It is included. A typical example of the position information is the coordinates in the display area of the display device 5.

Here, the designated operation will be specifically described with reference to FIG. FIG. 3 is a diagram showing an example of a spherical image displayed by the display device 5. More specifically, FIG. 3 is a diagram showing an example of displaying the spherical image 51 captured in the hoistway 21 on the display device 5 by a general spherical image viewer. In the figure, only a part of the spherical image 51 is displayed in the display area of the spherical image viewer, and the other parts are not displayed. In order to display the non-displayed portion, the spherical image 51 in the display area is usually slid according to the user's operation. A typical example of this operation is a drag operation using a mouse, which is an input device 4, with the cursor 52 in the display area of the spherical image viewer.

A typical example of a designated operation is a click operation using a mouse. Specifically, the user moves the cursor 52 on the spherical image 51 using a mouse, and performs a click operation at a position desired to be the starting point of a boundary that becomes a break when the spherical image 51 is developed. The designated operation is not limited to the click operation using the mouse.

The explanation will be given by returning to FIG. The boundary information generation unit 102 generates the above-mentioned boundary information in accordance with the generation instruction from the input control unit 101, and provides the boundary information to the boundary information acquisition unit 104. The boundary information generation unit 102 according to the present embodiment converts the coordinates in the display area of the display device 5 indicated by the position information included in the generation instruction into the corresponding coordinates in the spherical image 51. The boundary information generation unit 102 outputs the converted coordinates to the development unit 103 as boundary information. The boundary information is not limited to the coordinates as long as it can specify the position on the spherical image. Further, the boundary information generation unit 102 outputs the image information included in the generation instruction to the development unit 103 together with the boundary information.

The expansion unit 103 expands the spherical image into a plane image, and includes the boundary information acquisition unit 104. The boundary information acquisition unit 104 acquires boundary information and image information from the boundary information generation unit 102. When the boundary information acquisition unit 104 acquires the boundary information and the image information, the developing unit 103, when the spherical image indicated by the image information is a still image, together with the spherical image indicated by the image information. One or more spherical images different from the spherical image are read from the storage unit 11. Typical examples of another spherical image are (1) a spherical image stored in the same folder as the spherical image shown in the image information, and (2) a spherical image selected by the user. However, it is not limited to this example. On the other hand, when the spherical image indicated by the image information is one frame of the moving image, the developing unit 103 reads each frame of the moving image from the storage unit 11.

The developing unit 103 develops each of the plurality of spherical images read from the storage unit 11 into a flat image with the boundary defined by the boundary information as a break. At this time, the development unit 103 uniformly applies the boundary defined by one boundary information to the plurality of spherical images, thereby simultaneously developing each of the plurality of spherical images into a flat image. To do.

The boundary is a curve along the celestial sphere that passes through the coordinates indicated by the position information, with the zenith and nadir of the spherical image as end points. The zenith is the point where the line extending directly above the center of the celestial sphere intersects the celestial sphere. The nadir is the point where the line extending directly below the center of the celestial sphere intersects the celestial sphere. Then, since the distortion derived from the spherical image remains only when the spherical image is expanded on the plane image with the curve as the boundary, the developing unit 103 corrects so as to eliminate this distortion. As a result, the plane image becomes an image corresponding to the projection of the plane onto the screen. Then, the developing unit 103 stores the generated plurality of plane images in the storage unit 11. As for the technique of developing the spherical image into a plane image and the technique of correcting distortion, existing techniques can be used, and detailed description thereof will be omitted.

The display control unit 105 generates display data and causes the display device 5 to display the display data. When the display control unit 105 according to the present embodiment acquires the display instruction of the spherical image from the input control unit 101, the display control unit 105 reads the spherical image indicated by the display instruction from the storage unit 11 and reads the spherical image viewer. Is started and displayed on the display device 5. As a result, the display device 5 displays the spherical image 51 as shown in FIG.

Further, the display control unit 105 reads a plane image from the storage unit 11 and displays it on the display device 5. At this time, the display control unit 105 may activate the spherical image viewer to display the planar image, or may activate another image viewer to display the planar image. Further, the display control unit 105 may display the plane image by using the acquisition of the display instruction of the plane image from the input control unit 101 as a trigger, or triggers that the plane image is generated by the development unit 103. As a result, a flat image may be displayed. Further, the display control unit 105 may simultaneously display a plurality of plane images generated based on the same boundary information on the display device 5.

Here, a state of displaying a plane image will be specifically described with reference to FIG. FIG. 4 is a diagram showing an example of a flat image displayed on the display device 5 by a general image viewer. The example shown in the figure is an example in which the two spherical images are displayed as plane images 53a and 53b developed based on the same boundary information by the image viewer. The area inside the broken line on the plane image 53a is an area corresponding to the spherical image 51 displayed in the display area of the spherical image viewer shown in FIG. In the following, when the plane images 53a and 53b are not distinguished, they are simply referred to as "plane image 53".

As described above, the plane image 53 is an image obtained by developing a spherical image with a boundary defined by boundary information as a break. As shown in the figure, the display control unit 105 causes the display area of the image viewer to display the entire area of the plane image 53. As a result, the user can check the entire area at a glance and in an easy-to-see state without performing the slide operation that is normally performed when checking the spherical image with a general spherical image viewer. can do.

Each of the spherical images is an image taken by the spherical camera 2 installed in the car 22 at different times, and each of the plurality of planar images 53 is the whole sky based on the same boundary information. It is an image which developed each of the spherical images. Therefore, the portion having the same relative direction with respect to the spherical camera 2 at the time of each shooting is captured in the same region of each spherical image, and therefore, is captured in the same region of each planar image 53. For example, the side walls forming the hoistway 21 are reflected in the same area in each of the plane images 53 because the relative directions with respect to the spherical camera 2 do not change at the time of each shooting. Therefore, it is preferable that the display control unit 105 displays a plurality of plane images 53 to be displayed side by side so as to be scrollable in the vertical direction, as in the illustrated plane images 53a and 53b, from the viewpoint of improving visibility. With this display, for example, when it is desired to continuously check the side wall of the hoistway 21 across a plurality of plane images 53, the side wall can be easily checked by simply scrolling the screen in the vertical direction. Further, the area on the screen where the side wall of the user's attention is displayed does not change even if the screen is scrolled. That is, the user can continuously confirm the state of the side wall of the hoistway 21 to be confirmed without moving the line of sight significantly in the lateral direction.

In the above-mentioned example, the spherical image is expanded into a flat image in advance, but the spherical image is flattened by the display control unit 105 receiving the display instruction of the flat image as a trigger. It may be expanded to an image.

(Flow of image development processing)
Next, the flow of the image development process executed by the image processing device 1 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the flow of the image development process. FIG. 5A shows an example of the flow of the first half of the image development process, and FIG. 5B shows an example of the flow of the latter half of the image development process.

First, as shown in FIG. 5A, the communication unit 12 stores a plurality of spherical images acquired from the storage device 3 in the storage unit 11 (S1). Subsequently, when the input control unit 101 receives an operation signal for displaying any spherical image from the input device 4, the input control unit 101 outputs a display instruction of the spherical image to the display control unit 105. The display control unit 105 reads out the spherical image indicated by the acquired display instruction from the storage unit 11, activates the spherical image viewer, and displays the spherical image on the display device 5 (S2).

Subsequently, as shown in FIG. 5B, the input control unit 101 is in a state of waiting for an operation signal of a designated operation for designating a boundary to be a break in the spherical image (S3). When the input control unit 101 acquires an operation signal for the designated operation from the input device 4 (YES in S3), the input control unit 101 sends a generation instruction to the boundary information generation unit 102 to generate boundary information according to the acquired operation signal. Output.

Subsequently, the boundary information generation unit 102 generates boundary information that specifies a boundary passing through the position indicated by the position information included in the generation instruction (S4). Then, the boundary information acquisition unit 104 acquires the boundary information generated by the boundary information generation unit 102 and the image information included in the generation instruction (S5).

Subsequently, the development unit 103 reads out a plurality of spherical images including the spherical image indicated by the image information acquired together with the boundary information. Then, the developing unit 103 develops each of the read-out spherical spherical images into a plane image with the boundary defined by the boundary information as a break (S6). Then, the developing unit 103 stores the plane image in the storage unit 11 (S7).

Finally, the display control unit 105 reads a plane image from the storage unit 11 and displays it on the display device 5 (S8). This completes the image development process executed by the image processing device 1.

(Effect of image processing device 1)
As described above, according to the image processing device 1, a plurality of boundaries defined by the boundary information generated for one spherical image are uniformly applied to the plurality of spherical images. It is possible to develop each of the spherical images of the above into a plane image at once. Further, in order to generate the boundary information, the user only needs to specify the position that is the starting point of the boundary with respect to one spherical image. As a result, it is possible to reduce the complexity of individually designating boundaries for each of the plurality of spherical images.

In addition, as long as the position that is the starting point of the boundary is specified so that it does not overlap with the important points in inspection and abnormality detection, it is not necessary to adjust the shooting direction of the spherical camera 2 when shooting the spherical image. Therefore, it is possible to suppress the complexity of adjusting the shooting direction at the time of shooting.

[Embodiment 2]
Other embodiments of the present invention will be described in detail with reference to FIGS. 6 and 7. In each of the following embodiments, for convenience of explanation, the same reference numerals are added to the members having the same functions as the members described in the first embodiment, and the description thereof will be omitted.

In the present embodiment, the image processing device 1a that identifies the boundary that becomes the break of the spherical image by the image recognition process (image pattern recognition process) and automatically generates the boundary information will be described.

(Main part configuration of image processing device 1a)
First, the configuration of a main part of the image processing apparatus 1a will be described with reference to FIG. FIG. 6 is a block diagram showing an example of the main configuration of the image processing device 1a included in the image processing system 100a according to the present embodiment.

As shown in the figure, the image processing device 1a includes a control unit 10a, a storage unit 11, and a communication unit 12. As shown in the figure, the control unit 10a includes an input control unit 101, a boundary information generation unit 102a (first boundary information generation unit), a development unit 103, and a display control unit 105.

As described above, the image processing apparatus 1a according to the present embodiment automatically generates boundary information. Therefore, the input control unit 101 according to the present embodiment does not have to have a function of acquiring an operation signal of a designated operation and a function of outputting a generation instruction.

The boundary information generation unit 102a generates boundary information based on the result of image recognition processing for the spherical image. Specifically, the boundary information generation unit 102a first reads out a plurality of spherical images stored in the storage unit 11. The plurality of spherical images are the same as the plurality of spherical images read by the developing unit 103 in the first embodiment. Further, the trigger for the boundary information generation unit 102a to read the spherical image is not particularly limited. For example, the communication unit 12 may use the storage unit 11 to store the spherical image as a trigger.

Then, the boundary information generation unit 102a performs image recognition processing on any one of the read spherical images, and in advance based on the structural features of the hoistway 21 and the car 22 in the spherical images. Identify the part that meets the specified conditions. Examples of conditions include (1) the boundary between adjacent side walls in the hoistway 21, (2) the four corners of the car 22, (3) pillars along the elevating direction, and (4) the landing door and the side wall in the hoistway 21. The boundary with, etc. can be mentioned. The condition shall be specified in advance as a part that the user considers to be suitable for the boundary.

Then, the boundary information generation unit 102a generates boundary information with a portion satisfying the condition as a boundary. The boundary information generation unit 102a outputs the generated boundary information to the boundary information acquisition unit 104. Further, the boundary information generation unit 102a outputs the spherical image read from the storage unit 11 to the development unit 103.

(Flow of image development processing)
Next, the flow of the image development process executed by the image processing apparatus 1a will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of the flow of the image development process. The steps for executing the same processing as each step of the image development processing described in the first embodiment are assigned the same step numbers, and the description thereof will be omitted.

The boundary information generation unit 102a reads out a plurality of spherical images from the storage unit 11, performs image recognition processing on any one of the read spherical images, and identifies a portion that satisfies the condition (S11). .. Then, the boundary information generation unit 102a generates boundary information with a portion satisfying the condition as a boundary (S12). The boundary information generation unit 102a outputs the generated boundary information to the boundary information acquisition unit 104. Further, the boundary information generation unit 102a outputs the spherical image read from the storage unit 11 to the development unit 103.

(Effect of image processing device 1a)
As described above, according to the image processing apparatus 1a, it is possible to specify a portion suitable as a boundary in the spherical image and automatically generate boundary information with the specified portion as the boundary. As a result, the image processing device 1a can automatically set an appropriate boundary and develop the spherical image into a flat image with the set boundary as a break. Therefore, the user can confirm the developed plane image of the spherical image without performing the operation of setting the boundary.

The image processing device 1a may have a function of correcting the automatically generated boundary information according to the user's operation. Specifically, after the display control unit 105 displays the plane image on the display device 5, when the input control unit 101 acquires the operation signal of the designated operation (see the first embodiment), the input control unit 101 changes the boundary. The designation is output to the boundary information generation unit 102a. Then, the boundary information generation unit 102a newly generates boundary information based on the acquired boundary designation and outputs the boundary information to the expansion unit 103. As a result, the development unit 103 can generate a plane image with the corrected boundary. That is, when the user determines that the plane image displayed on the display device 5 is not appropriate, the user can modify the plane image to be considered to be suitable by his / her own operation.

[Embodiment 3]
Yet another embodiment of the present invention will be described in detail with reference to FIGS. 8-10. In the present embodiment, an image processing device 1b that expands one spherical image into a plurality of planar images while dividing one spherical image will be described instead of developing one spherical image into one planar image. A plane image developed while dividing a spherical image is hereinafter referred to as a "divided image".

(Main part configuration of image processing device 1b)
First, the configuration of a main part of the image processing apparatus 1b will be described with reference to FIG. FIG. 8 is a block diagram showing an example of the main configuration of the image processing device 1b included in the image processing system 100b according to the present embodiment.

The image processing device 1b includes a control unit 10b, a storage unit 11, and a communication unit 12 as shown in the figure. The control unit 10b, as shown in the figure, includes an input control unit 101, a boundary information generation unit 102, a development unit 103b, and a display control unit 105 (screen generation unit).

The image processing device 1b divides the spherical image into divided images located on a plane and corresponding to different directions. Since the general hoistway 21 has a rectangular parallelepiped shape and has four side walls, in the present embodiment, the image processing device 1b corresponds to the spherical image in four different directions located on a plane. It will be described as dividing into two divided images. Therefore, in the present embodiment, the designation operation is an operation for designating the positions of the four boundaries. Therefore, the input control unit 101 outputs a generation instruction including position information of each of the four positions based on the designated operation to the boundary information generation unit 102. In this embodiment, an example will be described in which each of the four designated positions corresponds to the position of the boundary between the adjacent side walls of the hoistway 21. More specifically, the four designated positions each correspond to the positions of the four boundaries on the hoistway 21.

The boundary information generation unit 102 generates four boundary information based on each of the four position information. Since the generation of each boundary information is the same as the method described in the first embodiment, the description thereof is omitted here.

The expansion unit 103b divides a plurality of spherical images read from the storage unit 11 into four by the boundary information acquisition unit 104, with the boundary defined by the four boundary information acquired from the boundary information generation unit 102 as a break. Expand while dividing into images. Then, the developing unit 103b stores the four divided images as one set in the storage unit 11.

Here, the division of the image will be specifically described with reference to FIG. FIG. 9A is a diagram showing boundaries for generating a divided image. Further, FIG. 9B is a diagram showing an example of a divided image displayed by the display device 5. The plane image 53a shown in FIG. 9A is a diagram in which the spherical image 51 is developed based on one boundary information described in the first embodiment. The development unit 103b of the present embodiment further divides the plane image 53a into four based on the three boundary information. These three boundary information are, for example, information that defines the boundary shown by the broken line shown in FIG. 9A. As a result, the plane image 53a becomes four divided images included in the set 54a of the divided images shown in FIG. 9B. As described above, since each of the four boundary information corresponds to the boundary between the adjacent side walls of the hoistway 21, the divided image is an image with each side wall of the hoistway 21 as the front.

The order of the division process and the expansion process is not limited. That is, the development unit 103b generates a plane image as shown in FIG. 9A from the spherical image, and then divides the plane image to divide the plane image as shown in FIG. 9B. It is not limited to the configuration that generates an image. For example, the development unit 103b may directly generate a divided image from the spherical image based on the four boundary information.

The display control unit 105 reads out the set of the divided images from the storage unit 11 and displays them on the display device 5. The display control unit 105 may display the divided image by using the acquisition of the display instruction of the divided image from the input control unit 101 as a trigger, or triggers that the divided image is generated by the developing unit 103b. As a result, the divided image may be displayed. Further, the display control unit 105 may display a set of a plurality of divided images generated based on the same boundary information on the display device 5 at the same time.

Here, a state of displaying a set of divided images will be specifically described with reference to FIG. 9B. Hereinafter, when the set 54a and 54b of the divided images are not distinguished, it is described as "the set 54 of the divided images".

It is preferable that the display control unit 105 displays the four divided images included in the divided image assembly 54 on the display device 5 side by side in the horizontal direction, as in the illustrated divided image assembly 54. As a result, the display control unit 105 can display the four divided images corresponding to the four directions side by side in the horizontal direction. For example, the display control unit 105 can display four divided images with each of the four side walls as the front side by side in the horizontal direction. As a result, the user can easily grasp the positional relationship of the areas reflected in each divided image, and can easily confirm the side wall of interest. Therefore, the hoistway 21 can be inspected and abnormality detection can be performed more easily.

Further, the display control unit 105 can display the set 54 of the plurality of divided images to be displayed on the display device 5 by arranging them so as to be scrollable in the vertical direction, as in the set 54a and 54b of the shown divided images. It is preferable from the viewpoint of improving visibility. At this time, it is preferable to display the divided images taken in the same direction side by side so as to be scrollable in the vertical direction. For example, the display control unit 105 displays a set of horizontally arranged images with each of the four side walls as the front side in the same horizontal arrangement order so as to be scrollable in the vertical direction. With this display, for example, when it is desired to continuously check the side walls of the hoistway 21, the screen can be easily checked by simply scrolling the screen in the vertical direction. Further, the area on the screen where the side wall of the user's attention is displayed does not change even if the screen is scrolled. That is, the user can continuously confirm the state of the side wall of the hoistway 21 to be confirmed without moving the line of sight significantly in the lateral direction.

In the above-mentioned example, the spherical image is expanded into the divided image in advance, but the spherical image is divided by the display control unit 105 receiving the display instruction of the divided image as a trigger. It may be expanded to an image.

(Flow of image development processing)
Next, the flow of the image development process executed by the image processing apparatus 1b will be described with reference to FIG. FIG. 10 is a flowchart showing an example of the flow of the image development process. FIG. 10A shows an example of the flow of the first half of the image development process, and FIG. 10B shows an example of the flow of the latter half of the image development process. The steps for executing the same processing as each step of the image development processing described in the first embodiment are assigned the same step numbers, and the description thereof will be omitted. Therefore, the description of (a) in FIG. 10 will be omitted.

As shown in FIG. 10B, the input control unit 101 is in a state of waiting for an operation signal of a designated operation for designating boundaries at four locations, which is a break in the spherical image (S21). When the input control unit 101 acquires an operation signal for the designated operation from the input device 4 (YES in S21), the input control unit 101 sends a generation instruction for generating boundary information to the boundary information generation unit 102 according to the acquired operation signal. Output.

The boundary information generation unit 102 generates four boundary information defining each of the four boundaries passing through the position indicated by the position information included in the generation instruction (S22). Then, the boundary information acquisition unit 104 acquires the four boundary information generated by the boundary information generation unit 102 and the image information included in the generation instruction (S23).

Subsequently, the developing unit 103b reads out a plurality of spherical images including the spherical image indicated by the image information acquired together with the boundary information. Then, the developing unit 103b develops each of the read-out spherical spherical images into a divided image with the boundary defined by the boundary information as a break (S24). Then, the developing unit 103b stores the four divided images divided from one spherical image as one set in the storage unit 11 (S25).

Finally, the display control unit 105 reads out the set of the divided images from the storage unit 11 and displays them on the display device 5 (S26). This completes the image development process executed by the image processing device 1b.

(Effect of image processing device 1b)
As described above, according to the image processing device 1b, each of the plurality of spherical images can be expanded into four divided images based on the four boundary information generated for one spherical image. it can. As a result, the user can confirm the situations of the four different orientations located on the plane in the hoistway 21 as separate images. For example, if the image processing device 1b uses each of the spherical images as a divided image with each side wall of the hoistway 21 as the front, the user confirms the situation of each side wall of the hoistway 21 as a separate image. Because it can be done, visibility is good. Therefore, the inspection of the hoistway 21 and the abnormality detection can be carried out more easily.

In the present embodiment, an example of expanding the spherical image into four divided images has been described, but the number of divided images is not limited to this example. Further, in the present embodiment, the example in which the position of the boundary is specified by the user has been described, but as in the second embodiment, the position of the boundary may be automatically specified by the image recognition process. Further, a configuration in which the position of the boundary is specified by the user and a configuration in which the position of the boundary is automatically specified may be combined.

In the case of a configuration in which the position of the boundary is automatically specified, the image processing device 1b generates boundary information by using a ratio (typically, an aspect ratio) such as the size of the hoistway 21 and the size of the car 22. May be good. For example, the spherical camera 2 is installed near the center of the outer upper part of the car 22 having a square upper surface, and the image processing device 1b is used to vertically and horizontally the upper surface of the car 22 (hereinafter, simply referred to as “upper surface”). The case where the information indicating the ratio is stored in the storage unit 11 will be described as an example. In this case, almost all of the upper surface of the car 22 except for the vicinity of the center is captured in the spherical image taken by the spherical camera 2. Therefore, the image processing device 1b only needs to identify one of the four corners of the car 22 by the image recognition process, and the spherical image processing device 1b is based on the information indicating the aspect ratio stored in the storage unit 11. The remaining three corners of the top surface in the image can be identified, and all boundary information for dividing the spherical image can be generated. As described above, according to this configuration, the boundary information necessary for generating the divided image can be generated by using the ratio such as the size of the hoistway 21 and the size of the car 22.

[Modification 1]
Modifications according to each embodiment of the present invention will be described in detail with reference to FIGS. 11 and 12. FIG. 11 is a schematic view showing another configuration example of the elevator 20. FIG. 12 is a schematic view showing still another configuration example of the elevator 20. Although the description will be given here as a modification of the first embodiment, the present modification is also applicable to the second and third embodiments.

In the above-described first embodiment, an example in which the spherical camera 2 is one has been described. However, the number of spherical cameras 2 is not limited to one. For example, as shown in FIG. 11, there are two spherical cameras 2: a spherical camera 2a installed on the outer upper part of the car 22 and a spherical camera 2b installed on the outer lower part of the car 22. It may be.

Further, for example, as shown in FIG. 12, there are three spherical cameras 2 in which the spherical camera 2a and the spherical camera 2b plus the spherical camera 2c installed in the pit 25 are added. May be good. The omnidirectional camera 2 may be one of the omnidirectional camera 2a and the omnidirectional camera 2b, and the omnidirectional camera 2c.

In the configuration in which the plurality of spherical cameras 2 are provided, each of the spherical cameras 2 may take a still image at the same timing (or take a moving image at the same timing). preferable.

Further, the storage unit 11 according to this modification may be configured to store a plurality of spherical images taken at the same time in association with each other. As a result, the development unit 103 according to the present modification can store each of the plane images developed from the plurality of spherical images in association with each other in the storage unit 11. Then, the display control unit 105 according to this modification may display the associated plane images on the display device 5 by arranging them in the vertical direction. As a result, the user can check at a glance the situation at the same time at different positions of the side wall of interest of the hoistway 21.

[Modification 2]
Other modifications according to each embodiment of the present invention will be described in detail. Although it will be described here as a modification of the first embodiment, this modification can also be applied to the second and third embodiments.

In the first embodiment described above, the boundary information generation unit 102 and the development unit 103 have been described as a configuration included in the control unit 10 of the image processing device 1. However, the boundary information generation unit 102 and the expansion unit 103 may be configured to be included in the spherical camera 2. That is, the spherical camera 2 may be configured to develop the generated spherical image into a flat image and store the flat image in the storage device 3. In this case, the image processing system 100 receives the operation signal from the input device 4 instead of the image processing device 1, reads out the plane image stored in the storage unit 11, and displays the plane image on the display device 5. Any configuration may include an information processing device having a function of causing the image.

[Example of realization by software]
The control blocks (particularly the control units 10, 10a, 10b) of the image processing devices 1, 1a, 1b may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by a CPU (hardware). It may be realized by software using a Central Processing Unit).

In the latter case, the image processing devices 1, 1a, and 1b are a CPU that executes a command of a program that is software that realizes each function, and a ROM in which the program and various data are readablely recorded by a computer (or CPU). It is equipped with a Read Only Memory) or a storage device (these are referred to as "recording media"), a RAM (Random Access Memory) for developing the above program, and the like. Then, the object of the present invention is achieved by the computer (or CPU) reading the program from the recording medium and executing the program. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1,1a, 1b Image processing device 2 Spherical camera (360 degree camera)
5 Display device 51 Spherical image (360 degree image)
53, 53a, 53b Plane image 102, 102b Boundary information generation unit (second boundary information generation unit)
102a Boundary information generation unit (first boundary information generation unit)
103, 103b Expansion unit 104 Boundary information acquisition unit 105 Display control unit (image display unit, screen generation unit)

Claims (5)

Boundary information acquisition unit that acquires boundary information that defines the boundary that is the break of the 360-degree image when the 360-degree image in the hoistway of the elevator , taken by the 360-degree camera, is expanded and displayed on a flat image. When,
By applying the boundary defined by the boundary information to a plurality of 360-degree images, a developing unit that expands each 360-degree image into a flat image, and
By image recognition processing on the image data of the 360-degree image, a portion that satisfies a predetermined condition based on the structural features of at least one of the hoistway and the car of the elevator is specified, and that portion is the boundary. An image processing device including a first boundary information generation unit that generates information and provides the information to the boundary information acquisition unit . The claim is characterized in that the boundary information acquisition unit acquires boundary information necessary for developing a 360-degree image while dividing it into four plane images corresponding to four different directions located on a plane. the image processing apparatus according to 1. The first boundary information generation unit identifies a portion satisfying the above conditions as one of the boundaries, and then identifies the remaining boundary based on at least one ratio of the hoistway size and the car size. , The image processing apparatus according to claim 2. The image processing apparatus according to claim 2 or 3 , further comprising a screen generation unit that generates a screen in which four plane images developed by the development unit are arranged side by side. The image processing apparatus according to claim 4 , wherein the screen generation unit generates a screen in which a set of horizontally arranged plane images corresponding to each 360-degree image is vertically arranged.

Images