JP2013092855A - Image processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus and program for achieving both the protection of a privacy region and its effective utilization in a picked-up image.SOLUTION: An information processing apparatus includes a region identifying unit to identify a privacy region in a picked-up image, and an image processing unit to insert, into the privacy region of the picked-up image, an image having different information from the image in the privacy region.

Description

  The present disclosure relates to an image processing device and a program.

  In recent years, processing and editing processing of captured images acquired by an imaging apparatus such as a digital still camera or a video camera have been widely performed. For example, Patent Document 1 discloses a technique for replacing a predetermined pattern existing in a background scene with an alternative pattern existing in the foreground scene when a composite video image is obtained by combining the background scene and the foreground scene. Yes.

  Patent Document 2 discloses a system for changing advertisement contents such as a signboard of a computer game or a signboard of a sports stadium broadcasted on TV. Patent Document 3 discloses a technique for detecting a specific area such as a signboard portion from a captured image and inserting an image showing other advertisement contents into the specific area, for example.

  Further, with the development of information communication technology, it is possible to easily provide captured images to an unspecified number of people via the Internet. For this reason, in order to protect privacy, special processing such as mosaic processing or black painting processing is often applied to a region related to privacy in a captured image. As is well known, special processing is applied to inappropriate images, confidential information, and the like, even for content broadcast on television or content recorded on an optical disc. Special processing such as mosaic processing and black painting processing is disclosed in, for example, Patent Document 4.

Japanese Translation of National Publication No. 06-510893 JP 2002-15223 A JP 2008-227813 A JP 2009-194687 A

  However, even if the privacy area can be protected by special processing such as mosaic processing or black painting processing, the information is lost or deteriorated in the specially processed privacy area. Visual information can hardly be obtained from the privacy area where is given.

  Therefore, the present disclosure proposes a new and improved image processing apparatus and program that can simultaneously realize protection and effective utilization of a privacy area in a captured image.

  According to the present disclosure, an area specifying unit that specifies a privacy area from a captured image, and an image processing unit that inserts an image having information different from the image in the privacy area into the privacy area of the captured image. An image processing apparatus is provided.

  Further, according to the present disclosure, the computer inserts an area specifying unit that specifies a privacy area from a captured image, and an image having information different from the image in the privacy area into the privacy area of the captured image. A processing unit and a program for functioning as the processing unit are provided.

  As described above, according to the present disclosure, it is possible to simultaneously protect and effectively use a privacy area in a captured image.

It is explanatory drawing which showed the structure of the image processing system by embodiment of this indication. It is explanatory drawing which showed the specific example of the captured image. It is the functional block diagram which showed the structure of the portable terminal 20-1 by 1st Embodiment. It is explanatory drawing which showed the specific example of the captured image after the image process by the image process part. It is the flowchart which showed operation | movement of the portable terminal 20-1 by 1st Embodiment. It is the sequence diagram which showed the modification of 1st Embodiment. It is the functional block diagram which showed the structure of the portable terminal 20-2 by 1st Embodiment. It is explanatory drawing which showed the specific example of the captured image containing a local area | region. It is explanatory drawing which showed the other specific example of the captured image containing a local area | region. It is explanatory drawing which showed the other specific example of the captured image containing a local area | region. It is explanatory drawing which showed the other specific example of the captured image containing a local area | region. It is explanatory drawing which showed the specific example of the captured image after the image process by the image process part 254. FIG. It is explanatory drawing which showed the other specific example of the captured image after the image process by the image process part. It is the flowchart which showed operation | movement of the portable terminal 20-2 by 2nd Embodiment. It is explanatory drawing which shows the specific example of the captured image in which code information was inserted. It is explanatory drawing which shows the specific example of the captured image in which code information was inserted. 3 is an explanatory diagram showing a hardware configuration of a mobile terminal 20. FIG.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

Moreover, this indication is demonstrated according to the item order shown below.
1. 1. Configuration of image processing system 2. Organize terms First embodiment 3-1. Background of the first embodiment 3-2. Configuration of portable terminal according to first embodiment 3-3. Operation of portable terminal according to first embodiment 3-4. Modified example 4. Second embodiment 4-1. Background of the second embodiment 4-2. Configuration of portable terminal according to second embodiment 4-3. 4. Operation of portable terminal according to second embodiment 5. Supplement to the first embodiment and the second embodiment Hardware configuration Conclusion

<1. Configuration of image processing system>
The technology according to the present disclosure can be implemented in various forms as will be described in detail in “2. First Embodiment” to “4. Third Embodiment” as an example. In the following, first, a basic configuration of an image processing system common to the respective embodiments will be described with reference to FIG.

  FIG. 1 is an explanatory diagram illustrating a configuration of an image processing system according to an embodiment of the present disclosure. As illustrated in FIG. 1, the image processing system according to the embodiment of the present disclosure includes an image providing server 10, a mobile terminal 20, a person recognition server 30, and an SNS server 40.

  The image providing server 10, the mobile terminal 20, the person recognition server 30, and the SNS server 40 are connected via the communication network 12 as shown in FIG. The communication network 12 is a wired or wireless transmission path for information transmitted from a device connected to the communication network 12. For example, the communication network 12 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various local area networks (LANs) including the Ethernet (registered trademark), a wide area network (WAN), and the like. . Further, the communication network 12 may include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

  The image providing server 10 provides a captured image acquired by the imaging device. For example, the image providing server 10 may store captured images at multiple points on the map, and may provide the stored captured images to the mobile terminal 20 in response to a request from the mobile terminal 20. Alternatively, the image providing server 10 may manage an individual or business operator's Web page and provide the mobile terminal 20 with a Web page including a captured image in response to a request from the mobile terminal 20.

  The portable terminal 20 is an image processing device that acquires a captured image and processes the acquired captured image. The portable terminal 20 can acquire a captured image by various methods. For example, the mobile terminal 20 may acquire a captured image from the image providing server 10 illustrated in FIG. 1, may acquire a captured image using the imaging function of the mobile terminal 20, or may reproduce the recording medium. A captured image may be acquired. Furthermore, the mobile terminal 20 may acquire a captured image by receiving a broadcast.

  In addition, in FIG. 1, although the smart phone is shown as an example of the portable terminal 20, the portable terminal 20 is not limited to a smart phone. For example, the mobile terminal 20 may be a mobile phone, a PHS (Personal Handyphone System), a portable music playback device, a portable video processing device, a portable game device, or the like. Furthermore, the portable terminal 20 is only an example of an image processing device according to the present disclosure. The functions of the image processing apparatus according to the present disclosure are performed by an information processing apparatus such as a PC (Personal Computer), a home video processing apparatus (DVD recorder, VCR, etc.), a PDA (Personal Digital Assistant), a home game machine, and a home appliance. Is also feasible. Furthermore, the image providing server 10 on the image providing side can also function as an image processing apparatus according to the present disclosure.

  The person recognition server 30 recognizes who the person is by analyzing the face image of the person. For example, the person recognition server 30 has a database for storing face feature amounts for each person, detects a face image from a captured image by face pattern matching, analyzes the feature amount of the face image, and corresponds to the analyzed feature amount Person recognition is performed by searching the database for the person to perform.

  An SNS (Social Networking Service) server 40 provides a service that supports a connection between people. For example, the SNS server 40 manages each user's Web page. A user can construct a new human relationship by publishing his / her profile information such as age, sex, hobby, origin, residence area, and diary (blog) on his Web page.

  The basic configuration of the image processing system according to the present disclosure has been described above. Subsequently, after organizing terms used in the description of the present embodiment, each embodiment of the present disclosure will be described in detail.

<2. Terminology>
Hereinafter, terms used in the description of the present embodiment will be described. However, it should be noted that the content shown below is merely an example of the meaning of each term, and in this application, each term may be used with a meaning different from the content shown below.

(Privacy / Privacy Area)
Privacy is a matter related to personal life and private matters. The privacy area is an area related to privacy in the captured image. Hereinafter, with reference to FIG. 2, privacy and a privacy area | region are demonstrated more concretely.

  FIG. 2 is an explanatory diagram showing a specific example of a captured image. The captured image shown in FIG. 2 includes a vehicle 51, houses 52 and 53, and a person 54. At the same time, this captured image includes many privacy and privacy regions.

  For example, since the vehicle 51 is identified by the license plate 61, the captured image including the vehicle 51 together with the license plate 61 indicates privacy that the user of the vehicle 51 was present at the imaging point of the captured image. For this reason, the license plate 61 corresponds to a privacy area because it is an area related to privacy indicating personal private life and private matters. In FIG. 2, a four-wheeled vehicle is shown as the vehicle 51, but the vehicle 51 may be a motorcycle, a large vehicle, or the like.

  In addition, as shown in FIG. 2, the captured image including the laundry 62 dried on the veranda of the house 52 indicates the privacy of clothes worn by the residents of the house 52. For this reason, since the laundry 62 is an area related to privacy indicating personal private life and private matters, the laundry 62 corresponds to a privacy area.

  As shown in FIG. 2, the captured image including the nameplate 63 of the house 53 shows privacy that the person indicated by the nameplate 63 lives in the house 53. For this reason, the nameplate 63 is an area related to privacy indicating personal private life and private matters, and therefore corresponds to the privacy area.

  Further, since a person can be recognized by a face, as shown in FIG. 2, the captured image including the person 54 together with the face 64 indicates privacy that the person 54 exists at the imaging point of the captured image. For this reason, the face 64 corresponds to a privacy area because it is an area related to privacy indicating personal private life and private matters.

(Special processing)
This is image processing for limiting the amount of visual information from a local area that constitutes a part of a captured image. Examples of special processing include mosaic processing, blurring processing, and painting processing. Such special processing is often applied to a privacy area such as a human face or a vehicle license plate. For this reason, it is also possible to treat a local area that has undergone special processing as a privacy area.

<3. First Embodiment>
The terms used in the description of the present embodiment have been described above. Subsequently, the first embodiment of the present disclosure will be described in detail.

[3-1. Background of First Embodiment]
With the development of information communication technology in the near future, it is possible to easily provide captured images to an unspecified number of people via, for example, the communication network 12. For this reason, in order to protect privacy, special processing such as mosaic processing or black painting processing is often applied to the privacy region in the captured image.

  However, even if the privacy area can be protected by special processing such as mosaic processing or black painting processing, the information is lost or deteriorated in the specially processed privacy area. Visual information can hardly be obtained from the privacy area where is given.

  Therefore, the first embodiment of the present disclosure has been created by focusing on the above circumstances. According to the first embodiment of the present disclosure, it is possible to simultaneously realize protection and effective utilization of a privacy area in a captured image. Hereinafter, the first embodiment of the present disclosure will be described in detail.

[3-2. Configuration of portable terminal according to first embodiment]
FIG. 3 is a functional block diagram showing the configuration of the portable terminal 20-1 according to the first embodiment. As illustrated in FIG. 3, the mobile terminal 20-1 according to the first embodiment includes a system controller 220, an imaging unit 224, a communication unit 228, a storage unit 232, an operation input unit 236, and a privacy area specification. Unit 240, insertion region setting unit 244, image acquisition unit 248, image processing unit 252, display control unit 256, and display unit 260.

(System controller)
The system controller 220 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and controls the overall operation of the mobile terminal 20-1. In FIG. 3, functional blocks such as the privacy area specifying unit 240, the insertion area setting unit 244, the image processing unit 252, and the display control unit 256 are shown separately from the system controller 220. Functions such as the insertion area setting unit 244, the image processing unit 252, and the display control unit 256 may be realized by the system controller 220.

(Imaging part)
The imaging unit 224 images a subject and acquires a captured image. Specifically, the imaging unit 224 includes an imaging optical system such as an imaging lens and a zoom lens, an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and an imaging signal processing unit.

  The photographing optical system collects light emitted from the subject and forms a subject image on the imaging surface of the imaging element. The image sensor converts a subject image formed by the photographing optical system into an electrical image signal. The imaging signal processing unit includes a sample hold / AGC (Automatic Gain Control) circuit that performs gain adjustment and waveform shaping on an image signal obtained by the imaging device, and a video A / D converter, and obtains a captured image as digital data. In addition, the imaging signal processing unit performs white balance processing, luminance processing, color signal processing, blur correction processing, and the like on the captured image data.

  Such an imaging unit 224 supplies the acquired captured image to the communication unit 228, the storage unit 232, the privacy region specifying unit 240, or the display control unit 256 under the control of the system controller 220. The imaging unit 224 controls on / off of an imaging operation, drive control of a zoom lens and a focus lens of an imaging optical system, control of sensitivity and a frame rate of an imaging element unit, an imaging signal based on an instruction from the system controller 220. It includes a function as an imaging control unit that performs parameter control of each process of the processing unit, setting of execution processing, and the like.

(Communication Department)
The communication unit 228 is an interface with an external device, and communicates with the external device wirelessly or by wire. For example, the communication unit 228 can communicate with the image providing server 10, the person recognition server 30, the SNS server, and the like via the communication network 12. Examples of the communication method of the communication unit 228 include a wireless LAN (Local Area Network) and LTE (Long Term Evolution).

(Storage section)
The storage unit 232 is used for storing various data. For example, the storage unit 232 stores a captured image that is a processing target according to the present embodiment, an insertion image to be inserted into the captured image, and an image that has been processed by the image processing unit 252 described below. The storage unit 232 performs recording / saving and reading of various data according to the control of the system controller 220.

  The storage unit 232 may be a storage medium such as a nonvolatile memory, a magnetic disk, an optical disk, and an MO (Magneto Optical) disk. Examples of the nonvolatile memory include a flash memory, an SD card, a micro SD card, a USB memory, an EEPROM (Electrically Erasable Programmable Programmable Memory), and an EPROM (Erasable Programmable ROM). Examples of the magnetic disk include a hard disk and a disk type magnetic disk. Examples of the optical disk include a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-Ray Disc (registered trademark)).

(Operation input section)
The operation input unit 236 is a configuration for a user to input an operation. The operation input unit 236 generates a signal corresponding to the user operation and supplies it to the system controller 220. The operation input unit 236 may be, for example, an operator such as a touch panel, a button, a switch, a lever, or a dial, a light receiving unit for infrared signals generated by a remote controller, or a receiving unit for radio signals. Furthermore, the operation input unit 236 may be a sensing device such as an acceleration sensor, an angular velocity sensor, a vibration sensor, or a pressure sensor.

(Privacy domain specific part)
The privacy area specifying unit 240 specifies the privacy area from the captured image. The captured image may be an image acquired by the imaging unit 224, an image read / reproduced from the storage unit 232, or an image acquired by the image acquisition unit 248 via the communication unit 228.

  In addition, as described in “2. Arrangement of terms”, various objects are assumed as privacy areas. For this reason, the privacy area | region specific part 240 specifies a privacy area | region with a different method for every object.

  For example, the privacy area specifying unit 240 may detect a face from a captured image by an existing face detection method using face pattern matching, and specify an area including the detected face as a privacy area.

  Moreover, the privacy area | region specific | specification part 240 may detect the rectangular area | region where a character is contained like a license plate of a vehicle or a nameplate of a house in a captured image, and may specify the detected rectangular area | region as a privacy area | region. For example, the privacy area specifying unit 240 detects a rectangular area whose outline is a rectangle, a trapezoid, and a parallelogram in the captured image. Furthermore, when the rectangular area is detected, the privacy area specifying unit 240 determines whether or not a character is included inside the rectangular area. Then, when a character is included inside the rectangular area, the privacy area specifying unit 240 specifies the rectangular area as the privacy area.

  Moreover, the privacy area | region specific part 240 may detect the area | region where the dried laundry is contained in a captured image, and may specify the detected area | region as a privacy area | region. Here, the clothes of the laundry are not worn by a person. In addition, a plurality of laundry clothes are often dried side by side. Therefore, the privacy region specifying unit 240 detects clothes from the captured image, and the clothes are not worn by a person, there are clothes adjacent to the clothes, and the adjacent clothes are not worn by the person. The area including these clothes may be specified as the privacy area.

(Insertion area setting part)
The insertion area setting unit 244 sets the privacy area specified by the privacy area specifying unit 240 as an insertion area for inserting an insertion image to be described later. The insertion area setting unit 244 may set the entire privacy area or a part of the privacy area as the insertion area.

(Image acquisition unit)
The image acquisition unit 248 acquires an image such as a captured image to be subjected to image processing according to the present embodiment and an inserted image to be inserted into the captured image. For example, the image acquisition unit 248 may acquire an image from the image providing server 10 via the communication unit 228 or may acquire an image by reproduction / reading from the storage unit 232.

(Image processing unit)
The image processing unit 252 inserts the insertion image into the insertion region in the captured image set by the insertion region setting unit 244. Here, the insertion may be a process of overwriting the inserted image on the image in the insertion area, or may be a process of replacing the image in the insertion area with the inserted image. The inserted image may be an image acquired by the imaging unit 224, an image read / reproduced from the storage unit 232, or an image acquired by the image acquisition unit 248 via the communication unit 228. The image processing unit 252 processes such an insertion image so as to match the shape and size of the insertion area, and inserts it into the insertion area.

  Further, the image used as the insertion image by the image processing unit 252 is information different from the image in the insertion area (privacy area), and may be related to the image in the insertion area.

  For example, when the image in the insertion area is a license plate, the insertion image may include, for example, vehicle manufacturer information or vehicle type information, and manufacturer and product advertisement information. Further, the inserted image may include link information to a purchase site for vehicles, vehicle-related products, and related parts. In this case, the user can connect the portable terminal 20-1 to the link destination by selecting the insertion image.

  Further, when the image in the insertion area is a nameplate, the insertion image may include, for example, real estate sales information and advertisement information of real estate agent and rehouse agent information. Further, the inserted image may include link information of a real estate purchase site or a home page of a real estate agent and a rehouse agent.

  Further, when the image in the insertion area is a laundry, the insertion image may include advertisement information of a cleaning shop and advertisement information of a clothing store. Further, the inserted image may include link information to a home page of a cleaning shop or a clothing store or a clothing purchase site.

  In addition, when the image in the insertion area is a human face, the insertion image may include advertisement information such as an esthetic, glasses, and contact lenses.

  In addition, when the image in the insertion area is a person's face, the insertion image may be information publicly related to the person. For example, when the image in the insertion area is a person's face, the portable terminal 20-1 requests the person recognition server 30 to recognize the person, and uses information published in the SNS server 40 regarding the recognized person. You may get it. In this case, profile information such as the age and sex of the person can be used as the inserted image. When the mobile terminal 20-1 has a person recognition function, the mobile terminal 20-1 may recognize a person having a face in the insertion area.

  Further, the image processing unit 252 may set the link information of the Web page on which the person publishes the blog as the inserted image. Since the person included in the captured image is present at the imaging position of the captured image, there is a possibility that matters related to the periphery of the captured position are published on the blog. For example, when the imaging position of the captured image is a tourist spot or a shopping street, the person may publish an opinion or opinion on the tourist spot or the shopping street on a blog. For this reason, it becomes possible to assist the user of the portable terminal 20-1 who wants to know the peripheral information of the imaging position of the captured image by setting the link information of the Web page as the insertion image as described above.

  In the above description, the advertisement information has been described as an example of the inserted image. However, when position information indicating the imaging position is set in the image file of the captured image, the image processing unit 252 performs the advertisement related to the object near the imaging position. Information may be preferentially used as an insertion image. Alternatively, when the mobile terminal 20-1 has a position estimation function, the image processing unit 252 may preferentially use the advertisement information related to the object in the vicinity of the current position of the mobile terminal 20-1 as the insertion image. With this configuration, it is possible to enhance the advertising information appealing effect for the user of the mobile terminal 20-1.

  Here, a specific example of the captured image after the image processing by the image processing unit 252 described above will be described with reference to FIG.

  FIG. 4 is an explanatory diagram showing a specific example of a captured image after image processing by the image processing unit 252. More specifically, FIG. 4 shows a state after the image processing by the image processing unit 252 of the captured image shown in FIG. As shown in FIG. 4, the images in the privacy areas 61 to 64 included in the captured image shown in FIG. 2 are replaced with the inserted images 71 to 74.

  For example, the license plate 61 of the vehicle 51, which is the privacy area, is replaced with an insertion image 71 including the advertisement information of the manufacturer of the vehicle 51 that “TOYOHA sale is in progress”, as shown in FIG. 4.

  In addition, as shown in FIG. 4, the laundry 62 in the house 52 is replaced with an insertion image 72 including the cleaning shop advertisement information “cleaning Sato OO station square”. Further, as shown in FIG. 4, the nameplate 63 of the house 53 is replaced with an insertion image 73 including real estate sales information “XX station square new property release!”.

  Further, as shown in FIG. 4, the face 64 of the person 54 is replaced with an insertion image 74 including guidance information to the Web page “Click here for the blog!”. Note that link information is set in the inserted image 74, and when the inserted image 74 is selected by the user, the portable terminal 20-1 can access the linked Web page. Note that the image processing unit 252 may replace the face 64 of the person 54 with a pseudo-face image according to profile information such as age and sex. According to such a configuration, it is possible to grasp what kind of person tends to pass near the imaging position of the captured image from the captured image after image processing.

  In addition, the use of the captured image after the image processing by the image processing unit 252 described above is not particularly limited. For example, the captured image after image processing may be displayed on the display unit 260, stored in the storage unit 232, or transmitted to an external device via the communication unit 228.

  Moreover, since the advertising efficiency is poor when the same advertisement information is included in the same image, the image processing unit 252 inserts a different insertion image in each privacy area when the captured image includes a plurality of privacy areas (insertion areas). May be. However, when the number of types of inserted images is less than the number of privacy areas, the image processing unit 252 may exceptionally insert the same inserted image into different privacy areas, Special processing such as painting may be performed.

(Display control unit)
The display control unit 256 drives the pixel drive circuit of the display unit 260 based on the control of the system controller 220 and causes the display unit 260 to display an image. For example, the display control unit 256 may display the captured image after the image processing by the image processing unit 252 on the display unit 260. The display control unit 256 can perform, for example, brightness level adjustment, color correction, contrast adjustment, sharpness (outline emphasis) adjustment, and the like for image display. Also, an image such as enlarged image generation in which a part of the image data is enlarged or reduced image generation, soft focus, luminance inversion, highlight display (highlight display) of a part of the image, change in the overall color atmosphere, etc. It is also possible to perform effect processing, separation and synthesis of images for divided display of captured images.

(Display section)
The display unit 260 includes a pixel drive circuit, and drives the pixel drive circuit according to the control of the display control unit 256 to display an image. The display unit 260 may be a liquid crystal display or an organic EL display.

[3-3. Operation of portable terminal according to first embodiment]
The configuration of the mobile terminal 20-1 according to the first embodiment has been described above. Next, with reference to FIG. 5, the operation of the mobile terminal 20-1 according to the first embodiment will be described.

  FIG. 5 is a flowchart showing the operation of the portable terminal 20-1 according to the first embodiment. As shown in FIG. 5, first, when the mobile terminal 20-1 acquires a captured image (S110), the privacy area specifying unit 240 determines whether or not there is a privacy area in the captured image (S120). Note that the mobile terminal 20-1 may acquire a captured image by the imaging unit 224, may acquire a captured image by reading / reproducing the storage unit 232, or may acquire the captured image via the communication unit 228. You may get it.

  If there is a privacy area in the captured image, the privacy area specifying unit 240 specifies a privacy area such as a person's face or license plate (S130), and the insertion area setting unit 244 is specified by the privacy area specifying unit 240. The set privacy area is set as an insertion area for inserting an insertion image (S140).

  Subsequently, the image processing unit 252 acquires an insertion image related to the image in the insertion area, and processes the insertion image as necessary (S150). Then, the image processing unit 252 inserts the insertion image into the insertion region set by the insertion region setting unit 244, that is, replaces the image in the insertion region with the insertion image (S160).

  Thereafter, the portable terminal 20-1 repeats the processes of S130 to S160 until no privacy area is detected from the captured image (S120). When the privacy region is no longer detected from the captured image, the display control unit 256 causes the display unit 260 to display the captured image in which the privacy region is replaced with the insertion image by the image processing unit 252 (S170).

  Note that FIG. 5 shows an example in which the processing of S130 to S160 is repeated for each privacy area, but all privacy areas are first identified, and processing related to all privacy areas is performed in each step. Later, you may move on to the next step.

  As described above, according to the first embodiment of the present disclosure, it is possible to protect the privacy area included in the captured image and simultaneously insert an image including useful information in the privacy area. With this configuration, it is possible to effectively use the privacy area in the captured image and improve the added value of the image.

[3-4. Modified example]
In the above description, the example in which the mobile terminal 20-1 specifies the privacy region and inserts an image has been described. However, the above-described processing can be realized by cooperation of a plurality of devices. Hereinafter, as a modification, an example will be described in which the image providing server 10 and the portable terminal 20-1 cooperate to realize the processing of the first embodiment.

  FIG. 6 is a sequence diagram showing a modification of the first embodiment. As shown in FIG. 6, in the modification, the image providing server 10 specifies a privacy area in the captured image (S310), and sets the privacy area as the insertion area of the inserted image (S320). Then, the image providing server 10 transmits the captured image and the setting information of the insertion area to the portable terminal 20-1 (S330).

  Thereafter, for example, the image acquisition unit 248 of the portable terminal 20-1 acquires the insertion image, and the image processing unit 252 processes the insertion image as necessary. Then, the image processing unit 252 inserts the insertion image in the insertion area, that is, replaces the image in the insertion area with the insertion image (S350).

  In this way, it is possible to specify the privacy area and insert the inserted image using different devices. In this case, the privacy area in the captured image is effectively used and the added value of the image is improved. can do.

<4. Second Embodiment>
Heretofore, the first embodiment of the present disclosure has been described. Subsequently, a second embodiment of the present disclosure will be described.

[4-1. Background of Second Embodiment]
As described as the background of the first embodiment, in the near future, in order to protect privacy, special processing such as mosaic processing and black painting processing is often performed on the privacy area in the captured image. As is well known, special processing is applied to inappropriate images, confidential information, and the like, even for content broadcast on television or content recorded on an optical disc.

  However, even if the privacy information can be protected by special processing such as mosaic processing or black painting processing, the local area that has been subjected to special processing has lost or deteriorated information. Visual information is hardly obtained from the processed local area. For this reason, the captured image subjected to special processing has poor information transmission efficiency.

  Therefore, the second embodiment of the present disclosure has been created by focusing on the above circumstances. According to the second embodiment of the present disclosure, it is possible to improve information transmission efficiency, added value, convenience, and the like of a captured image subjected to special processing. Hereinafter, the second embodiment of the present disclosure will be described in detail.

[4-2. Configuration of portable terminal according to second embodiment]
FIG. 7 is a functional block diagram showing the configuration of the mobile terminal 20-2 according to the first embodiment. As illustrated in FIG. 7, the mobile terminal 20-2 according to the second embodiment includes a system controller 220, an imaging unit 224, a communication unit 228, a storage unit 232, an operation input unit 236, and a local area specification. Unit 242, insertion region setting unit 246, image acquisition unit 248, image processing unit 254, display control unit 256, and display unit 260. The configurations of the system controller 220, the imaging unit 224, the communication unit 228, the storage unit 232, the operation input unit 236, the image acquisition unit 248, the display control unit 256, and the display unit 260 are as described in the first embodiment. Detailed description will be omitted here.

(Local area identification part)
The local region specifying unit 242 specifies a local region that has been subjected to special processing that limits the amount of visual information from the captured image. Here, as described in “2. Arrangement of terms”, various image processing processes are assumed as special processing. For this reason, the local region specifying unit 242 specifies a local region that has been subjected to special processing using a different method depending on the type of special processing such as mosaic processing, blurring processing, and painting processing. Hereinafter, a more specific description will be given with reference to FIGS. 8 and 9.

  FIG. 8 is an explanatory diagram showing a specific example of a captured image including a local region. The captured image illustrated in FIG. 8 includes a plurality of local regions 81 to 84 that have been subjected to special processing. For example, the local area 81 corresponds to the area of the license plate of the vehicle 51, but since the blurring process is performed, it is difficult to determine the contents of the license plate. Similarly, the local regions 82 to 84 are also subjected to blurring processing, and it is difficult to determine the content of the image before processing.

  The local region specifying unit 242 may specify the local region subjected to such blurring processing by analyzing the frequency component of each region of the captured image. For example, since it is considered that the high-frequency component is low in the region subjected to the blurring process, the local region specifying unit 242 may specify a region having a low high-frequency component as compared with the surrounding area as the local region.

  9 and 10 are explanatory diagrams illustrating other specific examples of the captured image including the local region. A local region 85 included in the captured image illustrated in FIG. 9 is a region of the line of sight of the person 55 that has been subjected to the filling process. Since the line of sight has high human identification, it is difficult to identify the person 55 from the image in which the line of sight is filled. Similarly, the captured image shown in FIG. 10 includes local regions 81 ′ to 84 ′. Since these local regions 81 ′ to 84 ′ are filled, it is difficult to visually recognize information such as the original nameplate and number plate.

  The local region specifying unit 242 may specify the local region that has been subjected to such painting processing by detecting the edge component of the captured image. In addition, since the filling process is often performed on a rectangular region, the local region specifying unit 242 specifies a region in the rectangle as a local region when the contour shape specified by the detection of the edge component is a rectangle. May be.

  FIG. 11 is an explanatory diagram illustrating another specific example of a captured image including a local region. The captured image shown in FIG. 11 includes local regions 81 ″ to 84 ″. Since these local areas 81 ″ to 84 ″ are subjected to mosaic processing, it is difficult to visually recognize information such as the original nameplate and number plate. The local region to which such mosaic processing has been performed is a set of pixel blocks, and the pixels constituting the same pixel block are considered to have the same component. Therefore, the local region specifying unit 242 may specify a set of pixel blocks having the same component as the local region.

(Insertion area setting part)
The insertion area setting unit 246 sets the local area specified by the local area specifying unit 242 as an insertion area for inserting an insertion image to be described later. Note that the insertion area setting unit 244 may set the whole or a part of the local area as the insertion area.

(Image processing unit)
The image processing unit 254 inserts the insertion image into the insertion region in the captured image set by the insertion region setting unit 246. Here, the insertion may be a process of overwriting the inserted image on the image in the insertion area, or may be a process of replacing the image in the insertion area with the inserted image. The inserted image may be an image acquired by the imaging unit 224, an image read / reproduced from the storage unit 232, or an image acquired by the image acquisition unit 248 via the communication unit 228. The image processing unit 254 processes such an insertion image so as to match the shape and size of the insertion area, and inserts it into the insertion area.

  Further, the image used as the insertion image by the image processing unit 254 may be related to the peripheral image of the insertion region (local region). For example, when the peripheral image is a vehicle as in the local region 81 shown in FIG. 8, the inserted image may include, for example, vehicle manufacturer information or vehicle type information, and manufacturer and product advertisement information. Further, the inserted image may include link information to a purchase site for vehicles, vehicle-related products, and related parts. In this case, the user can connect the portable terminal 20-2 to the link destination by selecting the insertion image.

  Further, when the peripheral image is a house as in the local region 82 and the local region 83 illustrated in FIG. 8, the inserted image may include, for example, real estate sales information and advertisement information of real estate agent and rehouse agent information. Good. Further, the inserted image may include link information of a real estate purchase site or a home page of a real estate agent and a rehouse agent. Further, when the peripheral image is a person as in the local region 84 illustrated in FIG. 8, the inserted image may include advertisement information such as esthetics, glasses, and contact lenses.

  In the above description, the advertisement information has been described as an example of the inserted image. However, when position information indicating the imaging position is set in the image file of the captured image, the image processing unit 254 displays the advertisement related to the object near the imaging position. Information may be preferentially used as an insertion image. Alternatively, when the mobile terminal 20-2 has a position estimation function, the image processing unit 254 may preferentially use the advertisement information related to the object in the vicinity of the current position of the mobile terminal 20-2 as the insertion image. With this configuration, it is possible to increase the advertising information appeal effect for the user of the mobile terminal 20-2.

  Moreover, since the advertising efficiency is poor when the same advertisement information is included in the same image, the image processing unit 254 may insert a different insertion image in each local region when the captured image includes a plurality of local regions. . However, when the number of types of inserted images is less than the number of local regions, the image processing unit 254 may exceptionally insert the same inserted image into different local regions.

  Here, with reference to FIG. 12 and FIG. 13, a specific example of the captured image after the image processing by the image processing unit 254 described above will be described.

  FIG. 12 is an explanatory diagram showing a specific example of a captured image after image processing by the image processing unit 254. More specifically, FIG. 12 shows a state after the image processing by the image processing unit 254 of the captured image shown in FIG. 8, FIG. 10, and FIG. As illustrated in FIG. 12, the image processing unit 254 replaces the images in the local regions 81 to 84 included in the captured image illustrated in FIG. 8 and the like with the inserted images 91 to 74. Although FIG. 10 shows an example in which the image in the local area is replaced with the insertion image, the image processing unit 254 may insert the insertion image in a form that adds the insertion image to the local area. That is, the image processing unit 254 does not necessarily have to extract the original image in the local area.

  FIG. 13 is an explanatory diagram showing another specific example of the captured image after the image processing by the image processing unit 254. More specifically, FIG. 13 shows a state after the image processing by the image processing unit 254 of the captured image shown in FIG. As illustrated in FIG. 13, the image processing unit 254 can insert the insertion image 95 into the local region 85 that has been subjected to the filling process.

[4-3. Operation of portable terminal according to second embodiment]
The configuration of the mobile terminal 20-2 according to the second embodiment has been described above. Next, with reference to FIG. 14, the operation of the mobile terminal 20-2 according to the second embodiment will be described.

  FIG. 14 is a flowchart showing the operation of the mobile terminal 20-2 according to the second embodiment. As illustrated in FIG. 14, first, when the mobile terminal 20-2 acquires a captured image (S410), the local region specifying unit 242 determines whether the local region in which the amount of information is limited by special processing is in the captured image. It is determined whether or not (S420). Note that the mobile terminal 20-2 may acquire a captured image by the imaging unit 224, may acquire a captured image by reading / reproducing the storage unit 232, or may acquire the captured image via the communication unit 228. You may get it.

  If there is a local area in the captured image, the local area specifying unit 242 specifies a local area that has been subjected to special processing such as blurring or painting (S430), and the insertion area setting unit 246 The local area specified by the specifying unit 242 is set as an insertion area for inserting an insertion image (S440).

  Subsequently, the image processing unit 254 acquires an insertion image related to the image in the insertion area, and processes the insertion image as necessary (S450). Then, the image processing unit 254 inserts the insertion image into the insertion region set by the insertion region setting unit 246, that is, replaces the image in the insertion region with the insertion image (S460).

  Thereafter, the portable terminal 20-2 repeats the processes of S430 to S460 until no local region is detected from the captured image (S420). Then, when the local region is no longer detected from the captured image, the display control unit 256 causes the display unit 260 to display the captured image in which the insertion image is inserted into the local region by the image processing unit 254 (S470).

  Note that FIG. 14 shows an example in which the processing of S430 to S460 is repeated for each local region. However, first, all local regions are identified, and processing for all local regions is performed in each step. Later, you may move on to the next step. Further, as described as a modification in the first embodiment, the above-described processing can be realized by cooperation of a plurality of devices (for example, the mobile terminal 20-2 and the image providing server 10).

  In addition, the use of the captured image after the image processing by the image processing unit 254 described above is not particularly limited. For example, the captured image after image processing may be displayed on the display unit 260, stored in the storage unit 232, or transmitted to an external device via the communication unit 228.

  As described above, according to the second embodiment of the present disclosure, it is possible to insert an image including useful information in a region that has been subjected to special processing such as blurring processing or painting processing. With this configuration, it is possible to improve information transmission efficiency, added value, convenience, and the like of a captured image that has been subjected to special processing.

<5. Supplement to the first embodiment and the second embodiment>
In the above description, the image processing units 252 and 254 (hereinafter simply referred to as an image processing unit) describe an example in which an insertion image such as advertisement information is inserted into an insertion region such as a privacy region or a local region. It is not limited to the example mentioned above. For example, the inserted image may be code information such as an ISBN code, a POS code, a URL, a URI (Uniform Resource Identifier), an IRI (Internationalized Resource Identifier), a QR code, or a cyber code. Hereinafter, this point will be described more specifically.

  When an insertion area such as a privacy area or a local area is set, the image processing unit determines the code information to be inserted and the arrangement of the code information from the shape and size of the insertion area, and inserts barcode information in the insertion area. To do.

  FIG. 15 and FIG. 16 are explanatory diagrams showing specific examples of captured images in which code information is inserted. As illustrated in FIG. 15, the image processing unit may insert the QR codes 102 to 104 into the insertion region such as the privacy region or the local region, or may insert the URL 101. Further, since the insertion area in the vehicle 51 has a horizontally long shape, the image processing unit may insert two lines of URLs 101 into the insertion area in the vehicle 51 as shown in FIG. In addition, as illustrated in FIG. 16, the image processing unit determines an arrangement in which five QR codes are arranged from the shape of the eye line region that is the insertion region in the person 55, and QR codes 105 </ b> A to 105 </ b> A are inserted in the insertion region in the person 55. 105E may be inserted.

  In the above description, an example in which the insertion area is a privacy area or a local area has been described. However, the insertion area is not limited to such an example. For example, the insertion area may be an area designated by human. Further, the code information may be encryption key information, and when the encryption key information is decrypted, the code information may disappear and the original image may be reproduced. The code information may be watermark information such as copyright management information. Moreover, the code information inserted into the insertion area may be related to the insertion area or an image around the insertion area.

  As described above, it is possible to increase the added value and convenience of an image by inserting code information into the insertion area. In addition, an inserted image in which link information is set like the inserted image 74 shown in FIG. 4 cannot be used when the image is printed. However, a two-dimensional barcode such as a QR code or a cyber code is This is useful in that it can be used even when printed.

<6. Hardware configuration>
The above-described image processing by the mobile terminal 20 is realized in cooperation with hardware included in the mobile terminal 20 described below with reference to FIG.

  FIG. 17 is an explanatory diagram showing a hardware configuration of the mobile terminal 20. As shown in FIG. 13, the mobile terminal 20 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an input device 208, an output device 210, A storage device 211, a drive 212, an imaging device 213, and a communication device 215 are provided.

  The CPU 201 functions as an arithmetic processing device and a control device, and controls the overall operation in the portable terminal 20 according to various programs. Further, the CPU 201 may be a microprocessor. The ROM 202 stores programs used by the CPU 201, calculation parameters, and the like. The RAM 203 temporarily stores programs used in the execution of the CPU 201, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus including a CPU bus.

  The input device 208 includes input means for a user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 201. Etc. The user of the mobile terminal 20 can input various data and instruct processing operations to the mobile terminal 20 by operating the input device 208. This input device corresponds to the operation input unit 236 shown in FIGS. 3 and 7.

  The output device 210 includes a display device such as a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, and a lamp. Furthermore, the output device 210 includes an audio output device such as a speaker and headphones. For example, the display device displays a captured image or a generated image. On the other hand, the audio output device converts audio data or the like into audio and outputs it.

  The storage device 211 is a data storage device configured as an example of a storage unit of the mobile terminal 20 according to the present embodiment. The storage device 211 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 211 stores programs executed by the CPU 201 and various data. The storage device 211 corresponds to the storage unit 232 shown in FIGS.

  The drive 212 is a storage medium reader / writer, and is built in or externally attached to the portable terminal 20. The drive 212 reads information recorded on a removable storage medium 24 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 203. The drive 212 can also write information to the removable storage medium 24.

  The imaging device 213 includes an imaging optical system such as a photographing lens and a zoom lens that collects light, and a signal conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging optical system collects light emitted from the subject and forms a subject image in the signal conversion unit, and the signal conversion element converts the formed subject image into an electrical image signal. The imaging device 213 corresponds to the imaging unit 234 illustrated in FIGS. 3 and 7.

  The communication device 215 is a communication interface configured with, for example, a communication device for connecting to the communication network 12. The communication device 215 may be a wireless LAN (Local Area Network) compatible communication device, an LTE (Long Term Evolution) compatible communication device, or a wire communication device that performs wired communication. The communication device 215 corresponds to the communication unit 228 illustrated in FIGS. 3 and 7.

<7. Conclusion>
As described above, according to the first embodiment of the present disclosure, it is possible to protect the privacy area included in the captured image and simultaneously insert an image including useful information in the privacy area. With this configuration, it is possible to effectively use the privacy area in the captured image and improve the added value of the image.

  Further, according to the second embodiment of the present disclosure, it is possible to insert an image including useful information in an area that has been subjected to special processing such as blurring processing or painting processing. With this configuration, it is possible to improve information transmission efficiency, added value, convenience, and the like of a captured image that has been subjected to special processing.

  In addition, although preferred embodiment of this indication was described in detail, referring an accompanying drawing, the technical scope of this indication is not limited to this example. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  For example, each step in the processing of the portable terminal 20 of the present specification does not necessarily have to be processed in time series in the order described as a sequence diagram or a flowchart. For example, each step in the processing of the mobile terminal 20 may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as the CPU 201, the ROM 202, and the RAM 203 built in the mobile terminal 20 to perform the same functions as the components of the mobile terminal 20 described above. A storage medium storing the computer program is also provided.

The following configurations also belong to the technical scope of the present disclosure.
(1)
An area specifying unit for specifying a privacy area from the captured image;
An image processing unit for inserting an image having information different from the image in the privacy area into the privacy area of the captured image;
An image processing apparatus comprising:
(2)
The image processing apparatus according to (1), wherein the image processing unit inserts an image having relevance with an image in the privacy area into the privacy area.
(3)
The image processing apparatus according to (1) or (2), wherein the area specifying unit specifies a face area of a person included in the captured image as the privacy area.
(4)
The image processing apparatus includes:
An acquisition unit for acquiring information disclosed about a person recognized by image analysis of the face area identified as the privacy area;
Further comprising
The image processing apparatus according to (3), wherein the image processing unit inserts an image having information acquired by the acquisition unit into the face area of the person.
(5)
The image processing device according to any one of (1) to (4), wherein when the captured image includes a plurality of privacy regions, the image processing unit inserts a different image into each privacy region.
(6)
The image processing unit performs special processing that restricts the amount of visual information locally on some privacy areas when the number of types of images to be inserted is less than the number of privacy areas in the captured image. The image processing apparatus according to (5).
(7)
The image processing apparatus according to any one of (1) to (6), wherein the area specifying unit specifies a license plate of a vehicle included in the captured image as the privacy area.
(8)
The image processing apparatus according to any one of (1) to (7), wherein the area specifying unit specifies a nameplate of a building included in the captured image as the privacy area.
(9)
The image processing apparatus according to any one of (1) to (8), wherein the area specifying unit specifies clothes that are dried as laundry included in the captured image as the privacy area.
(10)
The image processing apparatus according to any one of (1) to (9), wherein the image inserted into the privacy area is an advertisement image.
(11)
The image processing apparatus according to any one of (1) to (10), wherein the image inserted into the privacy area includes access information to a network.
(12)
The image processing apparatus according to any one of (1) to (11), wherein the image inserted into the privacy area includes code information.
(13)
In any one of (1) to (12), the area specifying unit specifies, as the privacy area, a local area that has been subjected to special processing that limits a visual information amount in the captured image. The image processing apparatus described.
(14)
Computer
An area specifying unit for specifying a privacy area from the captured image;
An image processing unit for inserting an image having information different from the image in the privacy area into the privacy area of the captured image;
Program to function as

DESCRIPTION OF SYMBOLS 10 Image provision server 12 Communication network 20 Portable terminal 30 Person recognition server 40 SNS server 220 System controller 224 Imaging part 228 Communication part 232 Storage part 234 Imaging part 236 Operation input part 240 Privacy area specific part 242 Local area specific part 244, 246 Insertion Area setting unit 248 Image acquisition unit 252, 254 Image processing unit 256 Display control unit 260 Display unit

Claims (14)

An area specifying unit for specifying a privacy area from the captured image;
An image processing unit for inserting an image having information different from the image in the privacy area into the privacy area of the captured image;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the image processing unit inserts an image having relevance with an image in the privacy area into the privacy area.   The image processing apparatus according to claim 1, wherein the area specifying unit specifies a face area of a person included in the captured image as the privacy area. The image processing apparatus includes:
An acquisition unit for acquiring information disclosed about a person recognized by image analysis of the face area identified as the privacy area;
Further comprising
The image processing apparatus according to claim 3, wherein the image processing unit inserts an image having information acquired by the acquisition unit into the face area of the person.   The image processing device according to claim 1, wherein when the captured image includes a plurality of privacy areas, the image processing unit inserts a different image into each privacy area.   The image processing unit performs special processing that restricts the amount of visual information locally on some privacy areas when the number of types of images to be inserted is less than the number of privacy areas in the captured image. The image processing apparatus according to claim 5.   The image processing apparatus according to claim 1, wherein the area specifying unit specifies a vehicle license plate included in the captured image as the privacy area.   The image processing apparatus according to claim 1, wherein the area specifying unit specifies a nameplate of a building included in the captured image as the privacy area.   The image processing apparatus according to claim 1, wherein the area specifying unit specifies clothes that are dried as laundry included in the captured image as the privacy area.   The image processing apparatus according to claim 1, wherein the image inserted into the privacy area is an advertisement image.   The image processing apparatus according to claim 1, wherein the image inserted into the privacy area includes access information to a network.   The image processing apparatus according to claim 1, wherein the image inserted into the privacy area includes code information.   The image processing apparatus according to claim 1, wherein the region specifying unit specifies, as the privacy region, a local region that has been subjected to special processing that limits a visual information amount in the captured image. Computer
An area specifying unit for specifying a privacy area from the captured image;
An image processing unit for inserting an image having information different from the image in the privacy area into the privacy area of the captured image;
Program to function as

Images