JP7494573B2 - Image processing device, image processing system, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and a program.

Image capture devices that use multiple wide-angle or fisheye lenses to capture images in all directions are known. When capturing images using such an image capture device, there are cases where parts that you do not want to show are captured in the image. A technique is known that performs post-processing on the captured image to remove the images of the parts that you do not want to show.

For example, Patent Document 1 describes a method for setting a cropping range when generating a planar image from a panoramic image by detecting unnecessary areas in the panoramic image and excluding the detected unnecessary areas to specify the cropping range.

However, with conventional methods, when the area to be processed, such as an unnecessary area shown in an image, changes depending on the condition of the subject captured by the imaging device, there is an issue in that the area to be processed cannot be set with high accuracy.

In order to solve the above-mentioned problems, the invention of claim 1 is an image processing device comprising an image input means for receiving an input of an image, a calculation means for calculating the position of a predetermined pattern shown in the input image, an estimation means for estimating vertices that complement a specific shape formed based on the calculated positions of a plurality of predetermined patterns, an area setting means for setting an area to be processed based on the calculated positions of a plurality of predetermined patterns and the estimated vertices, an image processing means for performing image processing on the set area , and a storage means for storing position information indicating the position of the predetermined pattern in the input image, wherein the area setting means sets the area based on the calculated positions of the plurality of predetermined patterns and at least a part of the stored position information .

The present invention has the effect of improving the accuracy of recognition of areas that are subject to image processing on captured images.

FIG. 1 is a diagram illustrating an example of an image processing system according to an embodiment. 10A and 10B are diagrams for explaining an example of a configuration for setting a mask processing region according to the embodiment; 13 is a diagram for explaining another example of the configuration for setting a mask processing region according to the embodiment. FIG. 5A to 5D are schematic diagrams for explaining an example of mask processing in the image processing system according to the embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to an embodiment. 1A is a conceptual diagram illustrating an example of a data structure of each image and a data flow of image processing in generating a celestial sphere image; FIG. 1B is a conceptual diagram illustrating an example of the data structure of the celestial sphere image in a plan view; and FIG. 1C is a conceptual diagram illustrating an example of the data structure of the celestial sphere image in a spherical view. FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus according to an embodiment. FIG. 4 is a conceptual diagram illustrating an example of a position information management table according to the embodiment. 11 is a flowchart illustrating an example of mask processing using a reference format in the imaging device according to the embodiment. FIG. 2 is a schematic diagram illustrating an example of a reference format according to the embodiment. 10 is a flowchart showing an example of a process for setting a mask processing region in the imaging device according to the embodiment. 11A and 11B are schematic diagrams for explaining a process of setting a mask processing region according to the embodiment. FIG. 2A is a diagram showing an example of a captured image captured by the imaging device according to the embodiment, and FIG. 2B is a diagram showing an example of a processed image that has been subjected to mask processing by the imaging device according to the embodiment. FIG. 13 is a diagram showing another example of a processed image that has been subjected to a mask process by the imaging device according to the embodiment. 5A and 5B are schematic diagrams for explaining an example of mask processing in the image processing system according to the embodiment. 11 is a flowchart illustrating an example of a mask process using position information in the imaging device according to the embodiment. 5 is a schematic diagram showing an example of a detection position of a predetermined pattern according to the embodiment; FIG. FIG. 13 is a diagram for explaining an example of an outline of an image processing system according to a modified example of an embodiment. FIG. 13 is a diagram illustrating an example of a hardware configuration of a PC according to a modified example of the embodiment. FIG. 13 is a diagram illustrating an example of a functional configuration of an image processing system according to a modified example of an embodiment. FIG. 11 is a sequence diagram showing an example of a mask process for an imaging device in an image processing system according to a modified example of an embodiment.

Below, we will explain the mode for implementing the invention with reference to the drawings. Note that in the explanation of the drawings, the same elements are given the same reference numerals, and duplicate explanations will be omitted.

●Embodiment●
System Configuration Fig. 1 is a diagram showing an example of an image processing system according to an embodiment of the present invention. The image processing system shown in Fig. 1 is a system for performing image processing on a captured image obtained by capturing an image of a subject in a shooting space by an image capturing device 10.

The imaging device 10 captures an image of a subject in a shooting space such as a factory or a server room. In this shooting space, there are subjects to be photographed and objects to be concealed. The imaging device 10 is an image processing device that photographs a subject in the shooting space and performs masking processing on the captured image to conceal the objects to be concealed.

The imaging device 10 is a digital camera that can capture an image by capturing an object in a capture space. The imaging device 10 is, for example, a special digital camera for capturing a 360° panoramic image. The imaging device 10 may be a general digital camera (such as a single-lens reflex camera or a compact digital camera). In this embodiment, the imaging device 10 is described as a digital camera (special imaging device) for capturing a 360° panoramic image. The captured image may be a video or a still image, or may be both a video and a still image. The captured image may include sound together with the image. The imaging device 10 may be fixed, or may be held by a photographer or a mobile robot and capture images while moving.

Here, a configuration for setting a mask processing area, which is an area to be processed, will be described with reference to Figs. 2 and 3. Fig. 2 is a diagram for explaining an example of a configuration for setting a mask processing area according to an embodiment.

Figure 2 shows a configuration for concealing object J using LED devices 30A-30E. As shown in Figure 1, a subject (object J) that the photographer does not want to appear in the image exists within the shooting space. In this case, as shown in Figure 2, multiple LED devices 30A-30H (hereinafter referred to as LED device 30 when there is no need to distinguish between them) are arranged three-dimensionally around object J to be protected. In the example shown in Figure 2, eight LED devices 30A-30H are arranged.

The LED devices 30A-30H generate bright spots 35A-35H of a color as a predetermined pattern, respectively. These bright spots 35A-35H become a pattern for specifying a mask processing area. The LED device 30 is an example of a light-emitting device that generates bright spots of a predetermined color, and is a device that is placed by a photographer who intends to use the automatic mask processing function to specify a mask processing area in advance. It is desirable for the predetermined pattern to be a pattern that can be distinguished from single-color lamps such as red or green that are used in various electronic devices, and may be configured as a light intensity (blinking) pattern, a color of a bright spot, an arrangement of multiple bright spots, or a combination of these.

The LED device 30 is an example of a pattern generating means that is placed within the field of view of the imaging device 10 and generates a predetermined pattern. The pattern generating means may be a light emitting device that emits one or more bright spots. In addition to the LED device 30, the light emitting device may be, for example, a wire light or mesh light in which a light emitting element such as an LED is attached to a wire or mesh. The pattern generating means may be a stand type or ceiling type as shown in FIG. 2, or a patch type.

The imaging device 10 calculates the coordinate values of the central positions of bright spots 35A-35H that correspond to the light-emitting sites of multiple LED devices 30A-30H in the captured image. If the captured image is a video, comparing pixel values at the same coordinates between successive images will result in a difference in pixel values at coordinates corresponding to a blinking LED device 30 depending on whether it is on or off. The imaging device 10 can detect the position in the image that corresponds to the blinking LED device 30 by checking changes in pixel values, such as the light intensity (blinking) pattern of the bright spots 35.

The imaging device 10 can also identify the LED blinking pattern from the appearance pattern of an image showing a lit LED device 30 and an image showing an unlit LED device 30. By recording a specific blinking pattern in the imaging device 10, the imaging device 10 can detect that the specific blinking pattern is a predetermined pattern generated by the pattern generating means.

Figure 3 is a diagram for explaining another example of a configuration for setting a mask processing region according to the embodiment. The example shown in Figure 3 is a configuration in which a medium on which an image such as QR codes (registered trademark) 40A to 40H (hereinafter referred to as QR code 40 when there is no need to distinguish between them) is used instead of the LED device 30 as a pattern generating means for generating a predetermined pattern for identifying the region. The imaging device 10 can detect the predetermined pattern by registering specific character information that serves as an identifier for each of the multiple QR codes 40A to 40H and reading the QR codes 40A to 40H. The QR code 40 is installed as a printed or displayed medium. It is assumed that the imaging device 10 will move or change its orientation during shooting, so it is desirable to have multiple surfaces on which the QR code 40 can be read so that it can be recognized from multiple viewpoints.

The image as the predetermined pattern shown in FIG. 3 is not limited to a QR code, but may be a one-dimensional code, a two-dimensional code, or a specific image. The one-dimensional code is a so-called barcode. The two-dimensional code may be, for example, a matrix type or a stack type, and may be, in addition to the QR code, DataMatrix (DataCode), MaxiCode, or PDF417. The specific image is an image that includes numbers, codes, symbols, illustrations, or the like that can identify the predetermined pattern. Note that when an image is used as the predetermined pattern, the image captured by the imaging device 10 does not need to be a video and may be a still image.

In this way, in the image processing system, the photographer or the manager of the subject being photographed (the person in charge of the facility such as the resident of the room or the factory that accepts the photographer's photographs) physically and three-dimensionally arranges a pattern generating means that generates a predetermined pattern around the subject that the photographer wishes to conceal, so that the above-mentioned predetermined pattern appears in the captured image. This allows the photographer or manager to specify in advance, before photographing, the area that he or she wishes to exclude from the subject being photographed. Note that in the following explanation, an example will be given in which the predetermined pattern is bright spots 35 generated by the LED device 30, but the predetermined pattern may also be an image such as that shown in FIG. 3.

Overview Here, image processing in the image processing system according to the embodiment will be explained in outline with reference to Fig. 4. Fig. 4 is a schematic diagram for explaining an example of mask processing in the image processing system according to the embodiment. Note that Fig. 4 is a simplified explanation of the outline of image processing in the image processing system according to the embodiment, and details of functions and the like realized by the image processing system will be explained with reference to drawings and the like described later.

Figure 4 (A) shows a situation where multiple LED devices 30 are arranged as pattern generating means around object J, as shown in Figure 2. For example, a possible use is to send the captured image captured by the imaging device 10 to a remote location, allowing remote confirmation of on-site footage of a factory or server room. In particular, a method is conceivable in which the photographer walks around the site while taking video footage. In this case, since equipment in a specific location (e.g. object X) is highly confidential, a possible mechanism is to arrange pattern generating means such as LED devices 30 around object X, and perform image processing on the image of object J surrounded by a predetermined pattern such as bright spots 35 in the captured image.

However, as shown in FIG. 4(A), there are cases where one of the arranged LED devices 30 is hidden by an obstacle X. When one of the LED devices 30 is hidden, as shown in FIG. 4(B), the imaging device 10 is unable to detect the bright spot 35 generated by the hidden LED device 30, the object J falls outside the area to be processed, and the intended area cannot be hidden.

Therefore, as shown in FIG. 4(C), the imaging device 10 estimates the vertices that are missing to form the area to be processed from the shapes of the detected multiple bright points 35 (six points in this case). Then, as shown in FIG. 4(D), the imaging device 10 applies a mask processing area set by the detected multiple bright points 35 and the estimated vertices.

In conventional methods for hiding a specific image from a captured image, it is not possible to specify in advance the area to be hidden according to the shooting space captured by the imaging device, and it is not possible to set the processing area to be the subject of image processing in real time during shooting. In particular, as shown in FIG. 4(B), when an obstacle X or the like is present between the imaging device 10 and a predetermined pattern for identifying the area to be processed, it becomes impossible to accurately identify the area to be processed, and it is not possible to hide the area to be hidden.

Therefore, when the imaging device 10 uses a pattern generating means such as the LED device 30 to recognize a three-dimensionally enclosed area to be concealed, even if part of the area is not captured for some reason, the imaging device 10 can correctly recognize the area to be concealed by estimating the shape of the area to be processed, thereby improving the accuracy of mask processing for the area to be processed.

●Hardware Configuration Next, the hardware configuration of the imaging device 10 will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the hardware configuration of the imaging device according to the embodiment. Note that components may be added or deleted from the hardware configuration shown in FIG. 5 as necessary. In the following, the imaging device 10 is assumed to be an all-sphere (omnidirectional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and a normal digital camera, smartphone, or the like may be provided with substantially the same functions as the imaging device 10 by attaching an omnidirectional imaging unit to the device.

As shown in FIG. 5, the imaging device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input/output I/F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I/F 121.

Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I/F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input/output I/F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I/F 121 are also connected to the bus 110.

The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I/F bus, performs a predetermined process on each piece of image data, and then synthesizes the image data to create equirectangular projection image data.

The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using the I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b, and send it to the CPU 111.

The imaging control unit 105 also instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display. In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames/minute).

As described below, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I/F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in the middle of processing. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular projection image.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

The input/output I/F 116 is a general term for an interface circuit (such as a USB (Universal Serial Bus) I/F) with an external medium such as an SD card or a personal computer. The input/output I/F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input/output I/F 116, or transmitted to an external terminal (device) via the input/output I/F 116 as necessary.

The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi (registered trademark) via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

The electronic compass 118 calculates the direction of the imaging device 10 from the earth's magnetism and outputs direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes the shooting date and time of the image and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the imaging device 10. The angle change is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (imaging device 10) based on the acceleration detected by the acceleration sensor 120. The imaging device 10 improves the accuracy of image correction by providing the acceleration sensor 120. The network I/F 121 is an interface for performing data communication using a communication network such as the Internet via a router or the like.

○Spherical Image Here, the generation of a celestial sphere image and the generated celestial sphere image will be described with reference to FIG. 6. FIG. 6(A) is a conceptual diagram showing an example of the data structure of each image and the data flow of image processing in the generation of a celestial sphere image. First, the images directly captured by each of the image capture elements 103a and 103b are images that cover approximately a hemisphere of the celestial sphere in their field of view. Light incident on the lens 102 is imaged on the light receiving area of the image capture element 103 according to a predetermined projection method. The image captured here is captured by a two-dimensional image capture element whose light receiving area forms a planar area, and becomes image data expressed in a planar coordinate system. Typically, the obtained image is configured as a fisheye lens image including the entire image circle onto which each shooting range is projected, as shown by "partial image A" and "partial image B" in FIG. 6(A).

The partial images captured by the imaging elements 103a and 103b are corrected for distortion and synthesized to form a single spherical image. In the distortion correction and synthesis process, first, images including complementary hemispherical portions are generated from each partial image configured as a planar image. Then, the two images including the hemispherical portions are aligned (stitched) based on matching of overlapping areas, and the images are synthesized to generate a spherical image including the entire celestial sphere. Each hemispherical image includes an overlapping area with another image, but in the image synthesis, the overlapping areas are appropriately blended to create natural seams.

Fig. 6(B) is a conceptual diagram showing an example of the data structure of a celestial sphere image in a plane, and Fig. 6(C) is a conceptual diagram showing an example of the data structure of a celestial sphere image in a spherical surface. As shown in Fig. 6(B), image data of the celestial sphere image is expressed as an array of pixel values whose coordinates are a vertical angle φ made with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

As shown in FIG. 6C, each coordinate value (θ, φ) in the omnidirectional format is associated with each point on a sphere that represents all directions centered on the shooting point, and the all directions are associated with the omnidirectional image. The planar coordinates of the partial image captured with the fisheye lens and the coordinates on the spherical surface of the omnidirectional image are associated with each other in a predetermined conversion table. The conversion table is data that is created in advance by a manufacturer or the like according to a predetermined projection model based on the design data of each lens optical system, and is data that converts the partial image into a omnidirectional image.

In addition, the omnidirectional image may be subjected to either or both of zenith correction and rotation correction as appropriate, using information from electronic compass 118, gyro sensor 119, and acceleration sensor 120. Zenith correction here refers to a process of correcting a omnidirectional image that was actually taken with its central axis tilted relative to the direction of gravity to an omnidirectional image that appears as if it was taken with its central axis aligned with the direction of gravity (correction in the Roll and Pitch directions). Rotation correction refers to a process of further rotating the omnidirectional image that has been corrected by zenith correction so that its central axis is aligned with the direction of gravity, from a reference around the direction of gravity (correction in the Yaw direction).

In the following description, a configuration in which the imaging device 10 acquires a celestial sphere image will be described, but the target of the mask processing function may be an image captured by a normal camera. In addition, the above-mentioned zenith correction and rotation correction will be described as being performed before the captured image is input to the image input unit 15, but they may be performed after the mask processing has been applied. Furthermore, the above-mentioned imaging device 10 is an example of an imaging device capable of acquiring a wide-angle image, and a celestial sphere image is an example of a wide-angle image. Here, a wide-angle image is generally an image captured using a wide-angle lens, and is captured with a lens that can capture a wider range than the human eye can sense. Also, it generally means an image captured with a lens with a focal length of 35 mm or less in 35 mm film equivalent.

The above programs may be recorded in the form of installable or executable files on a computer-readable recording medium and distributed. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, SD cards, and USB memories. The recording media may be provided domestically or internationally as a program product. For example, the imaging device 10 realizes the image processing method according to the present invention by executing the program according to the present invention.

Functional Configuration Next, the functional configuration of the imaging device 10 according to the embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing an example of the functional configuration of the imaging device according to the embodiment. Note that Fig. 7 shows items related to the processing or operation described below.

The imaging device 10 has a communication unit 11, a reception unit 12, a display control unit 13, an imaging processing unit 14, an image input unit 15, a mask type determination unit 16, a detection unit 17, a calculation unit 18, a judgment unit 19, an estimation unit 21, a region setting unit 22, an image processing unit 23, an image output unit 24, and a storage/readout unit 29. Each of these units is a function or means realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 111 in accordance with a program for the imaging device that is expanded from the SRAM 113 onto the DRAM 114. The imaging device 10 also has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. 4.

The communication unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. The communication unit 11, for example, uses the network I/F 121 to perform data communication with other devices or terminals via a communication network. The communication unit 11 also uses, for example, the input/output I/F 116 to perform data communication with other devices or terminals via various cables, etc. The communication unit 11 also uses, for example, the short-range communication circuit 117 to perform data communication with other devices or terminals via short-range wireless communication technology.

The reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user. The display control unit 13 is mainly realized by the processing of the CPU 111, and displays various images. For example, the display control unit 13 performs image display in live view (a function for shooting while making it possible to check the shooting state in real time). During live view, the display control unit 13 controls display on an external display connected by a wired or wireless interface according to a standard such as HDMI (registered trademark) or Miracast (registered trademark) or on its own display means. The display control unit 13 can display an automatically detected mask processing area superimposed on the display of an input image on the live view screen. Note that the method of displaying the omnidirectional image is not particularly limited, and the omnidirectional image may be displayed as it is as a two-dimensional flat image, or the omnidirectional image may be pasted on a spherical object and displayed as a projected image when the spherical object is observed from a virtual camera. The photographer can confirm on the live view image that the subject to be protected has been masked.

The imaging processing unit 14 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, the imaging control unit 105, the microphone 108, and the sound processing unit 109, and captures a subject such as a landscape and acquires captured image data. The imaging processing unit 14 executes imaging processing, generates data of a celestial sphere image (for example, an equirectangular projection image), and inputs the celestial sphere image data to the image input unit 15. The celestial sphere image captured by the imaging processing unit 14 may be a single frame still image or a video consisting of multiple continuous frames. The imaging processing unit 14 outputs an actual captured image after a shooting instruction is given by pressing the operation unit 115 such as a shutter button, and can also output a real-time image for video confirmation before a shooting instruction is given.

The image input unit 15 is mainly realized by the processing of the CPU 111, and receives input of a captured image to be processed. The captured image to be subjected to mask processing is a spherical image. In the case of a still image, one frame of an image captured at a predetermined time point is input to the image input unit 15. In the case of a moving image, each image of a plurality of frames is input sequentially to the image input unit 15. In the case of still image shooting, each image of a plurality of frames may be input sequentially to the image input unit 15 in order to check the image before shooting.

The mask type determination unit 16 is mainly realized by the processing of the CPU 111, and determines the type of image processing to apply to the mask processing area.

The detection unit 17 is realized by the processing of the CPU 111, and detects a predetermined pattern generated by a pattern generating means such as the LED device 30 or the QR code 40, which is shown in the captured image input by the image input unit 15. For example, when the captured image is a still image, the predetermined pattern to be detected can be a one-dimensional code, a two-dimensional code, or a specific image or a pattern of bright spots in the image. For example, the color of the bright spots and the arrangement of multiple bright spots can be used alone or in combination as the pattern of the bright spots. Note that the pattern of the bright spots includes a single bright spot having a predetermined color. Furthermore, when the captured image is a moving image, the predetermined pattern to be detected includes a one-dimensional code, a two-dimensional code, or a specific image or pattern of bright spots in the image of each frame, as well as a temporal pattern of bright spots in an image sequence spanning multiple frames. Here, the temporal pattern of the bright spots is composed of a light intensity (blinking) pattern of a predetermined frequency of the bright spots across multiple frames, the color of the bright spots, the arrangement of multiple bright spots, or a combination of these. The arrangement of the bright spots can be graphically represented in a predetermined shape such as "○", "×", "□", or "▽".

Any known technology can be used to recognize one-dimensional codes, two-dimensional codes, and specific images, as well as bright spots and their temporal patterns in an image or image sequence. In addition, in order to recognize images in a spherical image, the imaging device 10 may perform a process of correcting distortion in areas in the image that correspond to one-dimensional codes and two-dimensional codes, as preprocessing, if necessary.

The calculation unit 18 is realized by the processing of the CPU 111, and calculates the position of the predetermined pattern detected by the detection unit 17. The calculation unit 18 calculates, for example, the position of the predetermined pattern in the image as a coordinate value on a spherical image format. The position of the predetermined pattern may be tracked across multiple frames to calculate the correspondence. The determination unit 19 is realized by the processing of the CPU 111, and performs various determinations.

The estimation unit 21 is realized by the processing of the CPU 111, and estimates vertices that complete a specific shape formed based on the positions of the multiple predetermined patterns detected by the detection unit 17. The estimation unit 21 estimates missing vertices to form a specific shape determined using the multiple detected predetermined patterns.

The region setting unit 22 is realized by the processing of the CPU 111, and sets a mask processing region (region to be processed) for the target image based on the position of the predetermined pattern detected by the detection unit 17 and the vertices estimated by the estimation unit 21. Here, the mask processing region includes a region that includes a polygon defined by connecting at least the positions of the predetermined patterns and the coordinate values of the estimated vertices, or a region that includes at least the positions of the predetermined patterns and the convex hull of the points (the smallest convex polygon that includes all given points) with the coordinate values of the estimated vertices as points. Note that the mask processing region may be set as a polygon or convex hull itself that connects the positions of the predetermined patterns and the coordinates of the estimated vertices, or may be any figure (including figures formed of any closed curve such as a circle or ellipse in addition to polygons) with a predetermined margin provided for the polygonal region. The mask processing region is obtained, for example, as an array of coordinate values (parameters may be included in the case of a curve or the like) on the spherical image format. The region setting unit 22 automatically recognizes the "surface" defined by the predetermined pattern and the estimated vertices, and sets the mask processing region for that surface.

The area setting unit 22 automatically recognizes the "surface" defined by the specified pattern and the estimated vertices, and sets a mask processing area for that surface. Note that the number of specified patterns detected by the detection unit 17 is preferably at least three or more, which is sufficient to define a surface, but is not necessarily limited to three or more, as long as it is possible to draw any two-dimensional figure with a specified margin.

The omnidirectional image includes a range of 360 degrees in the horizontal direction, and both ends of the omnidirectional image are connected to each other. Therefore, the mask processing area may straddle the end, and the mask processing area may be set separately at both ends of the omnidirectional image. In this case, the area setting unit 22 determines the intersection between the mask processing area and the end boundary, and sets the mask processing area including the intersection determined on each side.

Here, in addition to the coordinate values of the specific pattern, the detection unit 17 can also determine the group to which the detected specific pattern belongs. In this case, the region setting unit 22 can set different mask processing regions based on the identified group configuration. For example, the above-mentioned specific pattern can encode information using a one-dimensional code, a two-dimensional code, a bright spot pattern, or the like, so that multiple groups can be represented by encoded information (e.g., an identifier). Then, the region setting unit 22 can set one mask processing region for each of the one or more identified groups.

The image processing unit 23 is realized by the processing of the CPU 111, and performs image processing on an area corresponding to the mask processing area in the captured image input by the image input unit 15. The image processing is a masking process that irretrievably deletes or deteriorates image information in the masking process area in the image. The masking process is at least one of blurring, filling with a predetermined color or image pattern (checkered pattern, stripes, noise pattern (sandstorm), stamp, etc.), mosaic processing, and overwriting with a two-dimensional or three-dimensional graphic object. The masking process is applied to, for example, an image area of the omnidirectional image that corresponds to the masking process area on the omnidirectional image format.

The image output unit 24 is mainly realized by the processing of the CPU 111, and outputs the processed image processed by the image processing unit 23.

The storage/readout unit 29 is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads various data (or information) from the storage unit 1000. The storage unit 1000 also stores captured image data acquired by the imaging processing unit 14 or processed image data processed by the image processing unit 23. The image data stored in the storage unit 1000 may be configured to be deleted when a predetermined time has passed since it was stored, or data transmitted to another device or terminal may be deleted.

Position Information Management Table FIG. 8 is a conceptual diagram showing an example of a position information management table according to an embodiment. The position information management table is a table for managing position information indicating the position of a detected predetermined pattern. A position information management DB 1001 configured by a position information management table as shown in FIG. 8 is constructed in the storage unit 1000. This position information management table stores the positions of multiple predetermined patterns detected in an image (frame) previously input to the image input unit 15. In the position information management table, coordinates indicating the positions of multiple predetermined patterns are managed for each area recognized as the same by tracking.

●Processing or Operation of the Embodiment ○Mask Processing Using a Reference Format Next, the processing or operation of the image processing system according to the embodiment will be described with reference to Figs. 9 to 17. First, the mask processing using a reference format for setting a mask processing region will be described with reference to Figs. 9 to 14. Fig. 9 is a flowchart showing an example of mask processing using a reference format in an imaging device according to the embodiment. Note that the processing shown in Fig. 9 is processing when the image to be processed is a moving image, and the processing is repeated for each frame from the first frame to the last frame.

First, the imaging processing unit 14 of the imaging device 10 starts imaging processing for a predetermined area (step S11). Specifically, the photographer presses an input means such as a shutter button, and the reception unit 12 receives a request to start imaging. Then, in response to the request to start imaging received by the reception unit 12, the imaging processing unit 14 starts imaging processing.

Next, the mask type determination unit 16 determines the mask type to be applied to the captured image acquired by the imaging process in step S11 (step S12). The mask type is, for example, the type of image processing to be applied to the mask processing area, such as blurring or mosaic processing, and a predefined shape for identifying the mask processing area. The predefined shape is for determining the shape of the area to which mask processing is applied, and is, for example, a three-dimensional solid shape such as a rectangular parallelepiped or a triangular pyramid. This predefined shape corresponds to the arrangement shape of the pattern generating means, such as the LED device 30. The mask type is specified by the photographer or the like prior to the start of imaging in step S11, and is stored as a setting value in a register or the like. The mask type determination unit 16 reads out the stored setting value to determine the mask type to be applied.

Next, the image input unit 15 receives an input of the image of the current frame of the captured image acquired by the imaging processing by the imaging processing unit 14, and acquires the image to be processed (step S13). Note that when performing mask processing on a stored still image or video, rather than during shooting or live view, the image input unit 15 may receive an input of an image read out from the storage unit 1000, and acquire the image to be processed.

Next, the detection unit 17 detects a predetermined pattern shown in the image of the frame to be processed acquired by the image input unit 15 (step S14). As described above, the photographer who intends to capture a spherical image or the manager of the subject to be captured places a pattern generating means for generating a predetermined pattern around the subject to be hidden in advance. The detection unit 17 detects the predetermined pattern in the image capturing the periphery of the subject. Then, the calculation unit 18 calculates the position of the predetermined pattern detected by the detection unit 17 (step S15). When the pattern generating means is an LED device 30 as shown in FIG. 2, the calculation unit 18 calculates the coordinate value of the center position of the bright spot 35 corresponding to the light-emitting portion of each LED device 30 as the position of the predetermined pattern.

Next, the imaging device 10 executes a process of setting a mask processing area based on the position of the predetermined pattern calculated by the calculation unit 18 (step S16). Here, the process of step S16 will be described in detail with reference to Figs. 10 to 12.

First, the standard format indicating the default shape determined by the mask type determination unit 16 will be described with reference to FIG. 10. FIG. 10 is a schematic diagram showing an example of the standard format according to the embodiment. The predetermined pattern generated by the pattern generating means arranged to identify the mask processing area appears differently in the captured image depending on the shooting position and shooting direction of the imaging device 10. Therefore, as shown in FIG. 10, the imaging device 10 stores in advance a plurality of formats (format A to format D) in which the same shape is captured from different angles as standard formats indicating the default shape. FIG. 10 is an example of the standard format when a rectangular parallelepiped shape is selected as the default shape. The imaging device 10 executes the setting process of the mask processing area described later using the standard format corresponding to the default shape determined by the mask type determination unit 16. Note that the number and contents of the standard formats indicating the default shape are not limited to the example shown in FIG. 10. The standard format has a different format for each default shape.

Fig. 11 is a flowchart showing an example of a process for setting a mask processing area in an imaging device according to an embodiment. Fig. 11 shows an example in which the default shape determined as the mask type is a rectangular parallelepiped. The determination unit 19 determines a specific shape indicating the type of pre-stored standard format according to the number of predetermined patterns detected by the detection unit 17 and the positions of the predetermined patterns calculated by the calculation unit 18.

First, if the judgment unit 19 judges that a seven-point predetermined pattern has been detected (YES in step S161), it shifts the process to step S162. Then, the judgment unit 19 determines format D (seven diagonal points) shown in FIG. 10 as the specific shape from among the reference formats corresponding to the default shape determined by the mask type determination unit 16 (step S162). On the other hand, if the judgment unit 19 judges that the detected predetermined pattern is not seven points (NO in step S161), it shifts the process to step S163.

Next, if the judgment unit 19 judges that a six-point predetermined pattern has been detected (YES in step S163), it shifts the process to step S164. Then, if the judgment unit 19 judges that, among the standard formats corresponding to the default shape determined by the mask type determination unit 16, format B (six points diagonally horizontally) shown in FIG. 10 is applicable (YES in step S164), it determines format B (six points diagonally horizontally) as the specific shape and shifts the process to step S175. On the other hand, if the judgment unit 19 judges that format B (six points diagonally horizontally) is not applicable (NO in step S164), it shifts the process to step S165.

In step S165, if the determination unit 19 determines that format C (six vertical diagonal points) shown in FIG. 10 is applicable among the reference formats corresponding to the default shape determined by the mask type determination unit 16 (YES in step S165), it determines format C (six vertical diagonal points) as the specific shape and transitions the process to step S175. On the other hand, if the determination unit 19 determines that format C (six vertical diagonal points) is not applicable (NO in step S165), it transitions the process to step S166.

In step S166, the determination unit 19 determines format D (seven diagonal points) shown in FIG. 10 as a specific shape from among the reference formats corresponding to the default shape determined by the mask type determination unit 16. Then, in step S167, the estimation unit 21 estimates the vertices to be complemented to form format D (seven diagonal points), which is the determined specific shape (step S174).

FIG. 12 is a schematic diagram for explaining the process of estimating vertices that complement a specific shape according to the embodiment. In step S174, the predetermined pattern is in a state in which one of the seven vertices of format D is hidden as shown in FIG. 12. Here, a state in which vertex C is hidden by an obstacle is considered. In addition, it is assumed that the omnidirectional image is expressed using equirectangular projection and is captured with the imaging device 10 in an upright position.

In this case, points A and B, which are located horizontally in three-dimensional space, are located horizontally in the omnidirectional image represented by equirectangular projection. Points B and D, which are located vertically in three-dimensional space, are located vertically in the omnidirectional image represented by equirectangular projection. At this time, two horizontal sides (AC and EF) and two vertical sides (BD and FH) are detected in the omnidirectional image.

Since the hidden point C is located at a position vertical to point A, the position of point C can be found by knowing the length of side AC and the vertical relationship between points A and C. Here, for the upper surface ABFE and the lower surface CDHG, if the vertices constituting each surface are at approximately the same height, the vertical relationship between points A and C can be found from the height of point A and other points. Since the depth of the vertex position can be calculated in a spherical image, the length of side AC can also be calculated. In this way, the estimation unit 21 can estimate the position of point C, which is the vertex to be complemented, based on the positional relationship between the shape of format D and the specified pattern detected by the determination unit 19.

The method of estimating vertices that complement the specific shape shown in FIG. 12 is not limited to the method for a rectangular parallelepiped shape as described above, but may be a method that uses a standard format that corresponds to other shapes. In addition to the example shown in FIG. 12, existing geometric calculation methods can be applied to the method of estimating vertices that complement the specific shape.

Returning to FIG. 11, in step S163, if the judgment unit 19 judges that the detected predetermined pattern is not six points (NO in step S163), the process proceeds to step S167. Next, if the judgment unit 19 judges that the detected predetermined pattern is five points (YES in step S167), the process proceeds to step S168. If the judgment unit 19 judges that the format B (six points diagonally horizontally) shown in FIG. 10 corresponds to the standard format corresponding to the predetermined shape determined by the mask type determination unit 16 (YES in step S168), the judgment unit 19 determines the format B (six points diagonally horizontally) as the specific shape and proceeds to step S174. Then, the estimation unit 21 estimates the vertices to be supplemented to form the determined specific shape, format B (six points diagonally horizontally) (step S174). The process by the estimation unit 21 here is equivalent to the process described in FIG. 12, and estimates the positions of the vertices that are missing to form format B (six points diagonally horizontally).

On the other hand, if the determination unit 19 determines that format B (six points diagonally horizontally) does not apply (NO in step S168), the process proceeds to step S169. In step S169, the determination unit 19 determines format C (six points diagonally vertically) shown in FIG. 10 as the specific shape from among the reference formats corresponding to the default shape determined by the mask type determination unit 16. The estimation unit 21 then estimates the vertices to be supplemented to form format C (six points diagonally vertically), which is the determined specific shape (step S174). The process by the estimation unit 21 here is equivalent to the process described in FIG. 12, and estimates the positions of the vertices that are missing to form format C (six points diagonally vertically).

In step S167, if the judgment unit 19 judges that the detected predetermined pattern is not five points (NO in step S167), the process proceeds to step S170. Next, if the judgment unit 19 judges that a four-point predetermined pattern is detected (YES in step S170), the process proceeds to step S171. The judgment unit 19 determines the format A (four points on the front) shown in FIG. 10 as a specific shape among the standard formats corresponding to the default shape determined by the mask type determination unit 16. On the other hand, in step S170, if the judgment unit 19 judges that the detected predetermined pattern is not four points (NO in step S170), the process proceeds to step S172. In step S172, if the judgment unit 19 judges that a three-point predetermined pattern is detected (YES in step S172), the process proceeds to step S173. In step S173, the determination unit 19 determines format A (four front points) shown in FIG. 10 as a specific shape from among the reference formats corresponding to the default shape determined by the mask type determination unit 16. Then, the estimation unit 21 estimates vertices to be supplemented to form format A (four front points), which is the determined specific shape (step S174). The processing by the estimation unit 21 here is equivalent to the processing described in FIG. 12, and estimates the positions of vertices that are missing to form format A (four front points).

On the other hand, in step S172, if the judgment unit 19 judges that the detected predetermined pattern is not three points (NO in step S172), the process ends because there is no area that can be set using the detected predetermined pattern.

Then, the area setting unit 22 sets the specific shape formed by the position of the detected predetermined pattern and the vertices estimated in step S174 as the mask processing area (step S175). In this way, the imaging device 10 determines a specific shape corresponding to a standard format in which the predetermined pattern is captured differently from the number and positional relationship of the predetermined pattern, and estimates the vertices that complement the determined specific shape, thereby enabling the desired mask processing area to be set with high accuracy, for example, even when there are areas where the predetermined pattern is hidden by an obstacle or the like.

Returning to FIG. 9, the image processing unit 23 performs mask processing on the captured image acquired by the image input unit 15 (step S17). The image processing unit 23 performs mask processing of the mask type determined in step S12 on the mask processing area set in step S16. Specifically, the image processing unit 23 performs mask processing on the surfaces of the shape of the mask processing area set by the area setting unit 22. For example, in the case of a shape of format A (four points on the front), the image processing unit 23 performs mask processing on one surface connecting the four points, and in the case of format B (six points diagonally horizontally), the image processing unit 23 performs mask processing on two surfaces, the left and right surfaces. Similarly, in the case of format C (six points diagonally vertically), the image processing unit 23 performs mask processing on two surfaces, the top and bottom surfaces, and in the case of format D (seven points diagonally), the image processing unit 23 performs mask processing on three surfaces, the left, right and top surfaces.

Then, the image output unit 24 outputs the processed image processed by the image processing unit 23 (step S18). Specifically, the image output unit 24 outputs the processed image to, for example, the display control unit 13, and causes the display control unit 13 to display the processed image on the display. The image output unit 24 also outputs the processed image to, for example, the storage/readout unit 29, and causes the storage/readout unit 29 to store the processed image. Furthermore, the image output unit 24 outputs the processed image to the communication unit 11, and transmits the processed image to an external device via the communication unit 11. Note that the output of the processed image by the image output unit 24 may be any of the above processes. For example, the imaging device 10 may be configured to store the processed image to the storage/readout unit 29 as the output destination when shooting a video, and to display the processed image to the display control unit 13 as the output destination when checking the image during video shooting.

Now, the captured image and processed image acquired by the imaging device 10 will be described with reference to Figs. 13 and 14. Fig. 13(A) is a diagram showing an example of a captured image captured by the imaging device according to the embodiment. The captured image 200 shown in Fig. 13(A) includes a plurality of LED devices 30 as pattern generating means and bright spots 35 as a predetermined pattern generated from each LED device 30 around the object J to be processed. Also, the captured image 200 shows a state in which a part of the predetermined pattern (for example, bright spot 35C) is hidden by an obstacle X.

13B is a diagram showing an example of a processed image masked by the imaging device according to the embodiment. The processed image 300 shown in FIG. 13B is a state in which image processing has been performed on the mask processing area by the above-mentioned mask processing. The example of FIG. 13B illustrates a case in which black-painted mask processing is applied to the inside of a polygonal area connecting the coordinates of the vertices of the mask processing area. In the processed image 300, the surface of the mask processing area defined by connecting the center positions of the bright spots 35 corresponding to the light-emitting parts of the multiple LED devices 30 is painted black, and image information of the object J inside the mask processing area is removed. In this way, the imaging device 10 outputs a processed image in which the area defined by connecting the positions of multiple predetermined patterns present in the input captured image has been masked. The imaging device 10 can hide an object to be hidden from the captured image by applying mask processing such as blurring or mosaic processing along the surface included in the mask processing area.

Fig. 14 is a diagram showing another example of a processed image that has been subjected to mask processing by the imaging device according to the embodiment. The processed image 350 shown in Fig. 14 is in a state in which a black mask processing has been performed on the outside of the polygonal area connecting the coordinates of the vertices of the mask processing area. As in the processed image 350, the image processing unit 23 may be configured to perform mask processing on the area in the captured image 200 that is outside the mask processing area set by the area setting unit 22. Note that the method of applying the mask processing as shown in Figs. 13(B) and 14 may be a configuration that is selected when determining the mask type in step S12 of Fig. 9.

Then, if the image to be processed is the final frame (YES in step S19), the imaging device 10 ends the process. On the other hand, if the image to be processed is not the final frame (NO in step S19), the imaging device 10 repeats the process from step S12. Although the processing flow when shooting a video has been described above, if the captured image is a still image, the process may end without performing the process in step S109.

In this way, the imaging device 10 detects the predetermined pattern shown in the captured image, and determines a specific shape corresponding to the standard format determined based on the number and positions of the detected predetermined patterns. The imaging device 10 then estimates vertices that complement the determined specific shape, and performs mask processing on the area to be processed that is set using the positions of the detected predetermined patterns and the estimated vertices. As a result, even if part of the predetermined pattern for identifying the area to be processed is not shown in the captured image, the imaging device 10 can improve the recognition accuracy of the area to be processed by estimating the missing vertices from the shape of the pre-stored standard format.

In the above description, an example in which mask processing is performed on one mask processing area shown in the captured image has been shown, but the imaging device 10 may be configured to perform mask processing on multiple areas. In this case, the detection unit 17 detects different predetermined patterns by detecting, for example, differences in the colors of the bright spots 35 of the LED device 30 as identifiers. Then, the area setting unit 22 sets different mask processing areas for groups (areas) that include patterns having identifiers that satisfy a predetermined condition. Here, the predetermined condition is that the detected patterns have the same identifier, or that the identifier is within a predetermined numerical range. Note that the identifier for identifying the pattern is not limited to differences in the colors of the bright spots 35, but may also be differences in the blinking patterns of the bright spots 35 or coded information embedded in an image such as a QR code 40. As a result, the image processing system according to the embodiment can perform mask processing on multiple processing target areas by providing different patterns as predetermined patterns that can be detected by the imaging device 10.

Masking using position information Next, masking using the position information of the stored predetermined pattern will be described with reference to Figs. 15 to 17. Fig. 15 is a schematic diagram for explaining an example of masking in the image processing system according to the embodiment. As shown in Fig. 15(A), a predetermined pattern (e.g., a bright spot 35 generated by the LED device 30) for identifying the area to be processed is temporarily hidden due to the movement of a person or the like. In this case, the imaging device 10 estimates the position of the hidden predetermined pattern using position information indicating the position of the predetermined pattern acquired and stored during past shooting. Then, the imaging device 10 sets the area to be processed using the estimated position of the predetermined pattern, as in the above example, and performs masking on the area to be processed as shown in Fig. 15(B).

As a result, when continuous processing is performed, such as when shooting video, the imaging device 10 stores the position information of a specific pattern, and even if the specific pattern is temporarily hidden, the imaging device 10 can more stably determine the area to be processed by estimating the vertices to be complemented using at least a portion of the stored position information.

Figure 16 is a flowchart showing an example of mask processing using position information in an imaging device according to an embodiment. Note that the processing from step S31 to step S35 is similar to the processing from step S11 to step S15 shown in Figure 9, and therefore a description thereof will be omitted.

In step S36, if the position information indicating the position of the predetermined pattern is stored in the position information management DB 1001 (see FIG. 8) (YES in step S36), the imaging device 10 shifts the process to step S37. The determination unit 19 compares the position information stored in the position information management DB 1001 with the position of the predetermined pattern detected by the detection unit 17 (step S37). In this case, the determination unit 19 compares the number of stored position information with the number of detected predetermined patterns. Then, the estimation unit 21 estimates the vertices to be complemented to form a specific shape according to the comparison result in step S37 (step S38). If the number of predetermined patterns is less than the stored position information, the estimation unit 21 estimates the point of the stored position information that is farthest from the position of the detected predetermined pattern as the vertex to be complemented. This is because the detection of the predetermined pattern is repeated at short time intervals during video shooting, so that the position on the image does not change significantly from the previous process. FIG. 17 is a schematic diagram showing an example of the detection position of the predetermined pattern according to the embodiment. In the example of Figure 17, when compared with the position information shown in Figure 8, point F is the furthest from the detection position shown in Figure 17, so the estimation unit 21 estimates point F as the complementary vertex.

Then, the area setting unit 22 sets a mask processing area using the position of the predetermined pattern detected by the detection unit 17 and the vertices estimated in step S38 (step S39). The image processing unit 23 then performs mask processing on the set mask processing area (step S40). Furthermore, the image output unit 24 outputs the processed image that has been masked (step S41). The mask processing and the output processing of the processed image are the same as in the example described above.

Next, the storage/readout unit 29 stores the position information indicating the calculated position of the predetermined pattern in the position information management DB 1001 (step S41). The imaging device 10 uses the position information stored in step S41 for processing the next frame. Then, if the image to be processed is the final frame (YES in step S42), the imaging device 10 ends the processing. On the other hand, if the image to be processed is not the final frame (NO in step S42), the imaging device 10 repeats the processing from step S32.

On the other hand, in step S36, if the position information indicating the position of the specified pattern is not stored in the position information management DB 1001 (NO in step S36), the imaging device 10 transitions the process to step S39. In this case, since the imaging device 10 does not store position information and cannot use information from previous processing, the imaging device 10 performs processing from step S39 onwards using only the position of the specified pattern detected by the detection unit 17. Note that, when position information is not stored, the imaging device 10 may be configured to perform processing to set a mask processing area using the above-mentioned standard format (for example, the processing of FIG. 11).

In this way, even if the specified pattern temporarily becomes unrecognizable within the field of view, such as when the imaging device 10 is moved or an object moves and blocks the field of view of the imaging device 10, the imaging device 10 can respond to the temporary disappearance of the specified pattern by storing the position of the specified pattern once detected and using past information to infer the directional movement of the imaging field of view and estimate the position of the undetectable specified pattern.

Effect of the embodiment As described above, the image processing system according to the embodiment can improve the accuracy of mask processing for the area to be processed by recognizing a pre-specified mask processing area in an image and applying mask processing to the image.

In the above description, the omnidirectional image acquired by the imaging device 10 is input to the image input unit 15. However, the above masking process can be particularly suitably applied to wide-angle images (omnidirectional images, hemispherical images, panoramic images) acquired by the imaging device 10, which are different from normal cameras in which the photographer can easily remove undesirable objects from the shooting range by adjusting the shooting direction. In particular, when shooting with the imaging device 10, a wide range is the shooting range, and it is not easy to remove a subject to be protected from the shooting range, and the protected subject may also be dispersed over a wide range. By applying the above masking process to the omnidirectional image, the photographer can shoot the omnidirectional image with peace of mind. However, the above masking process itself may also be applied to images shot with a camera with a normal angle of view.

Also, from the perspective of the processing target, when the characteristics of the target to be protected are clear in advance, such as a human face or body, they can be recognized by face recognition, body recognition, or other object recognition technology. However, it may be difficult to define in advance the characteristics of the target to be protected. For example, in the case of content that provides a tour of the inside of a factory using spherical images, there may be a wide variety of subjects within the factory, and it is generally difficult to recognize such subjects in advance. Therefore, the masking process according to this embodiment can be particularly suitably applied to targets whose characteristics are difficult to define in advance.

Furthermore, in the case of a fixed camera (including cases where the camera orientation is variable) with a fixed installation position such as a surveillance camera, it is easy to specify a mask for a specific coordinate range on the image, and the masking process according to this embodiment is particularly suitable for images captured by an imaging device with an indefinite installation position such as a handheld camera or a camera fixed on a tripod.

The masking process according to this embodiment can also be suitably applied when performing real-time distribution using the imaging device 10. In particular, when distributing, there is a possibility that the imaging device 10 will be moved, and by using the masking process, it becomes possible to hide three-dimensional areas, allowing real-time distribution to be performed with peace of mind.

Modifications of the embodiment
Next, an image processing system according to a modified example of the embodiment will be described. The same configurations and functions as those of the above embodiment will be denoted by the same reference numerals, and the description thereof will be omitted. The image processing system according to the modified example of the embodiment is a system in which a mask process for a captured image acquired by an imaging device 10a is executed by a PC 50 connected to the imaging device 10a.

System Configuration Fig. 18 is a diagram for explaining an example of an outline of an image processing system according to a modified embodiment. As shown in Fig. 18, the image processing system according to the modified embodiment includes a PC 50 connected to an imaging device 10a.

The PC 50 is an image processing device used by a user to perform image processing on captured images. An editing application program for editing an image to be processed, such as a spherical image, or a control application program for controlling the imaging device 10a is installed in the PC 50. The PC 50 is also connected to the imaging device 10 via a cable such as a USB or a High-Definition Multimedia Interface (HDMI).

The imaging device 10a and the PC 50 may communicate wirelessly using a short-range wireless communication technology such as Bluetooth, NFC, or Wi-Fi without using the above-mentioned cable. The imaging device 10a and the PC 50 may also communicate via a communication network such as the Internet, a mobile communication network, or a LAN (Local Area Network), and the PC 50 may be located at a different location from the imaging device 10a. In this case, the communication network may include not only wired communication, but also parts built by wireless communication such as 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), Wi-Fi, or WiMAX (Worldwide Interoperability for Microwave Access). The communication network may also include a network built by blockchain.

The PC 50 may be a smartphone or a tablet terminal. The PC 50 may be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and assigned arbitrarily.

●Hardware configuration Fig. 19 is a diagram showing an example of the hardware configuration of a PC according to a modified embodiment. The PC 50 is constructed by a computer, and as shown in Fig. 19, includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I/F 516.

Of these, the CPU 501 controls the operation of the entire PC 50. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, character, or image. The display 506 is an example of a display unit. The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external device is, for example, the imaging device 10. The network I/F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 19.

The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 514 controls the reading and writing of various data from the DVD-RW 513, which is an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R or Blu-ray Disc, etc. The media I/F 516 controls the reading and writing (storing) of data from the recording medium 515, such as a flash memory.

The above programs may be distributed by recording them on a computer-readable recording medium in the form of an installable or executable file. Examples of recording media include CD-Rs, DVDs, Blu-ray Discs, SD cards, and USB memories. The recording media may also be provided domestically or internationally as a program product. For example, the PC 50 realizes the image processing method according to the present invention by executing the program according to the present invention.

Functional Configuration Fig. 20 is a diagram showing an example of the functional configuration of an image processing system according to a modified example of the embodiment. The imaging device 10a has a communication unit 11, a reception unit 12, a display control unit 13, an imaging processing unit 14, and a storage/readout unit 29. Each of these units has the same function as each unit of the imaging device 10 shown in Fig. 7.

The PC 50 has a communication unit 51, a reception unit 52, a display control unit 53, a mask type determination unit 54, a detection unit 55, a calculation unit 56, a judgment unit 57, an estimation unit 58, a region setting unit 59, an image processing unit 61, and a storage/readout unit 69. Each of these units is a function or a means for performing a function that is realized when any of the components shown in FIG. 19 operates in response to an instruction from the CPU 501 according to a PC program deployed on the RAM 503. The PC 50 also has a storage unit 5000 constructed from the ROM 502, HD 504, or recording media 515 shown in FIG. 19. A location information management DB 5001 is constructed in the storage unit 5000 and is configured from a location information management table such as that shown in FIG. 8.

The communication unit 51 is mainly realized by the processing of the CPU 501, and communicates various data or information with the imaging device 10a. The communication unit 51 uses, for example, the network I/F 509 to perform data communication with the imaging device 10a via a communication network. The communication unit 51 also uses, for example, the external device connection I/F 508 to perform data communication with the imaging device 10a via various cables, etc.

The reception unit 52 is mainly realized by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user. The display control unit 53 is mainly realized by the processing of the CPU 501, and causes various images to be displayed on the display 506. The reception unit 52 and the display control unit 53 have the same functions as the reception unit 12 and the display control unit 13 of the imaging device 10 shown in FIG. 7, respectively.

The mask type determination unit 54, the detection unit 55, the calculation unit 56, the judgment unit 57, the estimation unit 58, the region setting unit 59, and the image processing unit 61 are each realized by the processing of the CPU 501, and have the same functions as the mask type determination unit 16, the detection unit 17, the calculation unit 18, the judgment unit 19, the estimation unit 21, the region setting unit 22, and the image processing unit 23 of the imaging device 10 shown in FIG. 7. The storage/reading unit 69 is mainly realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000. The storage unit 5000 also stores captured image data acquired from the imaging device 10a or processed image data processed by the image processing unit 61. Note that the image data stored in the storage unit 5000 may be deleted when a predetermined time has passed since it was stored, or data transmitted to another device or terminal may be deleted.

Processing or operation of the modified example of the embodiment Next, processing or operation of the image processing system according to the modified example of the embodiment will be described with reference to Fig. 21. Fig. 21 is a sequence diagram showing an example of mask processing for an imaging device in the image processing system according to the modified example of the embodiment.

First, the imaging processing unit 14 of the imaging device 10a starts imaging processing for a specified area (step S51). Specifically, the photographer presses an input means such as a shutter button, and the reception unit 12 receives a request to start imaging. Then, in response to the request to start imaging received by the reception unit 12, the imaging processing unit 14 starts imaging processing. Next, the communication unit 11 of the imaging device 10a transmits the captured image data acquired in the imaging processing of step S51 to the PC 50 (step S52). As a result, the communication unit 51 of the PC 50 receives the captured image data transmitted from the imaging device 10a and receives input of the image to be processed.

The PC 50 performs image processing on the image input by the communication unit 51 (step S53). The image processing in step S53 is a mask processing similar to the processing shown in FIG. 9 or FIG. 16. In particular, the display control unit 53 of the PC 50 may display the input photographed image on the display 506 during the image processing in step S53, thereby allowing the user of the PC 50 to check the state of the photographed image. The reception unit 52 of the PC 50 may also receive a selection of processing contents for the photographed image or processed image displayed on the display 506. In this case, the reception unit 52 may receive a selection of a mask type by the user in the mask type determination processing shown in FIG. 9 and FIG. 15, for example, or may receive a selection of a correction of the mask type for the processed image that has been subjected to mask processing.

In this way, the image processing system according to the modified embodiment uses the PC 50 to perform image processing on the image acquired by the imaging device 10a, making it possible to check the captured image in real time and to perform corrections to the mask processing while checking the captured image or the processed image.

●Summary●
As described above, an image processing device (e.g., the imaging device 10 or the PC 50) according to an embodiment of the present invention receives an input of a captured image and calculates the position of a predetermined pattern (e.g., the bright spot 35 or the QR code 40) shown in the input captured image. The image processing device then estimates vertices that complement a specific shape formed based on the calculated positions of the plurality of predetermined patterns, sets a processing target area (e.g., a mask processing area) based on the calculated positions of the plurality of predetermined patterns and the estimated vertices, and performs image processing (e.g., mask processing) on the set area. In this way, the image processing device according to the embodiment can improve the recognition accuracy of the area that is the target of image processing on the captured image, and can also improve the accuracy of mask processing on the processing target area.

Furthermore, an image processing device according to one embodiment of the present invention (e.g., the imaging device 10 or the PC 50) determines a specific shape based on the calculated positions of a plurality of predetermined patterns (e.g., the bright spot 35 or the QR code 40), and estimates the vertices that are missing to form the determined specific shape. As a result, the image processing device according to the embodiment can improve the recognition accuracy of the area to be processed by estimating the vertices that are missing from the specific shape determined from the calculated positions of the predetermined pattern, even if part of the predetermined pattern for identifying the area to be processed is not captured in the captured image.

Furthermore, an image processing device according to an embodiment of the present invention (e.g., the imaging device 10 or the PC 50) stores position information indicating the position of a predetermined pattern (e.g., the bright spot 35 or the QR code 40) in an input captured image, and sets a processing target area (e.g., a mask processing area) based on the calculated positions of the multiple predetermined patterns and at least a part of the stored position information. As a result, even if a predetermined pattern cannot be recognized temporarily within the field of view, such as when the imaging device 10 is moved or an object moves and blocks the field of view of the imaging device 10, the image processing device according to the embodiment can respond to the temporary disappearance of the predetermined pattern by storing the position of the predetermined pattern once detected and using past information to infer the directional movement of the shooting field of view and estimate the position of the undetectable predetermined pattern.

Furthermore, an image processing device (e.g., imaging device 10 or PC 50) according to one embodiment of the present invention sets a processing target area (e.g., mask processing area) that includes at least one surface formed by the calculated positions of a plurality of predetermined patterns (e.g., bright spots 35 or QR codes 40) and the estimated vertices, and performs image processing (e.g., mask processing) on the surface included in the set area. In this way, the image processing device according to the embodiment can perform mask processing on a three-dimensional area that is to be hidden by performing mask processing along the surface included in the processing target area.

Furthermore, in an image processing device (e.g., the imaging device 10 or the PC 50) according to one embodiment of the present invention, the multiple predetermined patterns (e.g., the bright spots 35 or the QR codes 40) include different patterns. The image processing device then detects an identifier that identifies the pattern along with the predetermined pattern, and sets different areas for multiple groups having identifiers that satisfy predetermined conditions. In this way, the image processing device according to the embodiment can perform mask processing on multiple areas to be processed by detecting different patterns as the predetermined pattern.

In addition, the image processing device (e.g., PC 50) according to one embodiment of the present invention displays the input captured image on the display 506 (an example of a display unit) and accepts the selection of a type of image processing (e.g., mask processing) for the area to be processed (e.g., mask processing area). This allows the image processing device according to the embodiment to check the captured image in real time, and to make corrections to the processing while checking the captured image or the processed image.

Furthermore, the image processing system according to one embodiment of the present invention includes an image processing device (e.g., the imaging device 10 or the PC 50), and a plurality of pattern generating means (e.g., an LED device 30 or a medium on which an image such as a QR code 40 is printed or displayed) that are arranged within the field of view of the imaging means and generate a predetermined pattern. As a result, the image processing system according to the embodiment can specify in advance, by arranging the pattern generating means, an area to be excluded from the subject to be photographed before photographing.

●Additional Information●
Each function of the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU, and a conventional circuit module designed to execute each function described above.

The various tables in the above-described embodiments may be generated by the learning effect of machine learning, and by classifying the data of each associated item by machine learning, tables may not be used. Here, machine learning is a technology for enabling a computer to acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

So far, we have described an image processing device, an image processing system, an image processing method, and a program according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and can be modified within the scope of what a person skilled in the art can imagine, such as adding, modifying, or deleting other embodiments, and any aspect is within the scope of the present invention as long as it achieves the functions and effects of the present invention.

10 Imaging device (an example of an image processing device)
14 Imaging processing unit (an example of an imaging means)
15 Image input unit (an example of an image input means)
17 Detection unit (an example of a detection means)
18 Calculation unit (an example of a calculation means)
19 Judgment unit (an example of a decision means)
21 Estimation unit (an example of an estimation means)
22 Region setting unit (an example of a region setting means)
23 Image processing unit (an example of an image processing means)
30 LED device (an example of a pattern generating means)
40 QR code (an example of a pattern generating means)
50 PC (an example of an image processing device)
51 Communication unit (an example of an image input means)
52 Reception unit (an example of a reception means)
53 Display control unit (an example of a display control means)
55 Detection unit (an example of a detection means)
57 Judgment unit (an example of a decision means)
58 Estimation unit (an example of an estimation means)
59 Region setting unit (an example of a region setting means)
61 Image processing unit (an example of an image processing means)
506 Display (an example of a display unit)
1000 Storage unit (an example of a storage means)
5000 Memory unit (an example of a memory means)

JP 2016-133908 A

Claims (14)

an image input means for receiving an input of a captured image;
A calculation means for calculating a position of a predetermined pattern shown in the input captured image;
an estimation means for estimating vertices that complement a specific shape formed based on the calculated positions of the plurality of predetermined patterns;
a region setting means for setting a processing target region based on the calculated positions of a plurality of predetermined patterns and the estimated vertices;
an image processing means for performing image processing on the set area;
a storage means for storing position information indicating a position of a predetermined pattern in the input captured image;
Equipped with
The area setting means sets the area based on the calculated positions of a plurality of predetermined patterns and at least a part of the stored position information. 2. The image processing device according to claim 1, further comprising:
a determining means for determining the specific shape based on the calculated positions of a plurality of predetermined patterns;
The estimation means is an image processing device that estimates vertices that are missing for forming the determined specific shape. the area setting means sets the area including at least one surface formed by the calculated positions of the plurality of predetermined patterns and the estimated vertices;
3. The image processing apparatus according to claim 1 , wherein the image processing means performs image processing on a surface included in the set area. The image processing device according to claim 1 , wherein the predetermined pattern is a one-dimensional code, a two-dimensional code, or a specific image, or a pattern of bright spots or a temporal pattern of bright spots across a plurality of images. 5 . The image processing device according to claim 1 , wherein the region includes a region that includes a polygon defined by connecting the positions of the plurality of predetermined patterns and the vertices, or a region that includes a convex hull of the positions of the plurality of predetermined patterns and the vertices. The image processing device according to claim 1 , wherein the image processing is at least one of mask processing including blurring, filling with a predetermined color or image pattern, mosaic processing, and overwriting with a graphic object. The image processing device according to claim 1 , wherein the input captured image is a spherical image. The plurality of predetermined patterns includes different patterns, and further
a detection means for detecting a predetermined pattern shown in the input captured image,
The detection means detects an identifier that identifies a pattern,
The image processing device according to claim 1 , wherein the region setting means sets different regions for a plurality of groups having identifiers that satisfy a predetermined condition. The image processing device according to any one of claims 1 to 8 , further comprising:
an imaging means for acquiring the captured image,
The image input means is an image processing device that inputs the captured image. The image processing device according to any one of claims 1 to 8 , further comprising:
A display control means for displaying the input captured image on a display unit;
A reception means for receiving a selection of a type of image processing for the area;
An image processing device comprising: An image processing system comprising the image processing device according to any one of claims 1 to 8 ,
An imaging means for acquiring the captured image;
a plurality of pattern generating means arranged within a field of view of the imaging means and configured to generate the predetermined patterns;
An image processing system comprising: 12. The image processing system according to claim 11 , wherein the pattern generating means is a light emitting device that emits a bright spot, or a medium on which a one-dimensional code, a two-dimensional code, or a specific image is printed or displayed. An image processing method executed by an image processing device, comprising:
an image input step of receiving an input of a captured image;
A calculation step of calculating a position of a predetermined pattern shown in the input captured image;
an estimation step of estimating vertices that complement a specific shape formed based on the calculated positions of the plurality of predetermined patterns;
a region setting step of setting a processing target region based on the calculated positions of the plurality of predetermined patterns and the estimated vertices;
an image processing step for performing image processing on the set area;
a storage step of storing position information indicating a position of a predetermined pattern in the input captured image;
Run
The image processing method , wherein the area setting step sets the area based on the calculated positions of a plurality of predetermined patterns and at least a part of the stored position information. The image processing device includes:
an image input step of receiving an input of a captured image;
a calculation step of calculating a position of a predetermined pattern shown in the input captured image;
an estimation step of estimating vertices that complement a specific shape formed based on the calculated positions of the plurality of predetermined patterns;
a region setting step of setting a processing target region based on the calculated positions of the plurality of predetermined patterns and the estimated vertices;
an image processing step for performing image processing on the set area;
a storage step of storing position information indicating a position of a predetermined pattern in the input captured image;
Run the command,
The area setting step sets the area based on the calculated positions of a plurality of predetermined patterns and at least a part of the stored position information.