JP2017072945A - Image processor, image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique to correct a pick-up image to an image which is equivalent to an image picked up under a white light source while considering light from a display unit and light from an external light source.SOLUTION: The image processor performs series of processing: acquires a pick-up image of a subject; generates a piece of color correction information for color correction to cancel color characteristics based on the color characteristics of the image which is displayed on a display unit when picking up the image; and corrects the pick-up image based on the color correction information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique, and more particularly to a technique for correcting the color and gradation of an image.

  In recent years, with the widespread use of smartphones and tablet PCs equipped with various devices such as cameras, gravity sensors, gyro sensors, and displays, it has become possible to easily develop systems that combine these functions. For example, a health management system and a beauty management system can be constructed by periodically photographing and analyzing a face and an expression using a camera attached to the front of the display. Such a system is operated as a resident application, and a large number of analysis images can be obtained without restraining the user by photographing and collecting faces and facial expressions during operation of the smartphone. However, the lighting conditions are not always suitable for photographing in order to live daily life.

  To deal with such problems, prepare multiple correction map data for hue and gradation for appropriate correction in advance, and use the light source measurement / estimation result of the shooting situation and the subject detection result to obtain an appropriate correction map. The method of selecting and applying data is common.

  For example, Patent Literature 1 discloses a technique for correcting an image by measuring a light source in a shooting state. When photographing with a surveillance camera in a convenience store or the like, the follow / backlight to the light source of the subject moving within the angle of view changes due to the influence of a fixed object (window, obstacle, etc.) in the area within the angle of view of the camera. The technique disclosed in Patent Document 1 corrects captured images at a plurality of locations of the same subject tracked to a uniform color and gradation after measuring a light source for each region in the image from a plurality of captured images. It is. In this way, since the analysis accuracy using the captured image can be improved, for example, the accuracy of subject search using color as a condition after the fact can be increased. On the other hand, as a technique for obtaining a corrected image by actively controlling the light source itself, there is a technique disclosed in Patent Document 2. In this technique, continuous shooting is performed in synchronism with flashing a camera strobe in a predetermined time-series blinking change pattern, and these are combined to obtain a corrected image in which blurring is corrected.

WO20140806613 JP 2011-78074 A

  However, the techniques disclosed in both Patent Document 1 and Patent Document 2 are based on a light source that can be measured or controlled. Therefore, in the techniques disclosed in Patent Documents 1 and 2, for a situation where there is a light-emitting device that is difficult to predict (a display that varies depending on an arbitrary operation) other than a light source that can be measured or controlled in a dark place Image correction is difficult. For example, in photographing in a dark place such as before going to bed or at night, an appropriate amount of light cannot be obtained, and in addition, the display is affected by light emitted from a smartphone display or the like. A photographed image that is strongly influenced by light emission, such as a display, may not be sufficiently corrected because the hue and gradation are biased. If an image that is not sufficiently corrected is used in health management or beauty management, analysis accuracy is lowered.

  The present invention has been made in view of such problems, and a technique for correcting a captured image into an image equivalent to that under a white light source in consideration of light from a display unit, light from an external light source, and the like. I will provide a.

  One aspect of the present invention is an image processing apparatus, which obtains a color characteristic based on an acquisition unit that acquires a captured image of a subject and a color characteristic of an image displayed on a display unit when the captured image is captured. The image processing apparatus includes a generation unit that generates color correction information for performing color correction to cancel, and a correction unit that corrects the captured image based on the color correction information.

  According to the configuration of the present invention, the captured image can be corrected to an image equivalent to that under a white light source in consideration of light from the display unit, light from an external light source, and the like.

1 is a block diagram illustrating a configuration example of an image processing apparatus. The figure explaining image correction. The figure which shows a table group. 6 is a flowchart of processing performed by the image processing apparatus. 6 is a flowchart of processing performed by the image processing apparatus.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In this embodiment, a captured image of a subject is acquired, and color correction information for performing color correction that cancels the color characteristic is generated based on the color characteristic of the image displayed on the display unit when the captured image is captured. An example of an image processing apparatus that corrects a captured image based on color correction information will be described. First, a configuration example of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG.

  The arithmetic unit 101 executes various kinds of processing using computer programs and data stored in the RAM 108, thereby controlling the operation of the entire image processing apparatus 100 and each of which will be described later as what the image processing apparatus 100 performs. Execute or control processing.

  The display unit 102 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the calculation unit 101 with an image or a character. When the image processing apparatus 100 is a projection device such as a projector, a display unit that displays an image or character to be projected, and a projection unit that projects the image or character displayed on the display unit onto a screen or the like And having the following. As described above, the display unit 102 presents an image, characters, and the like to the observer.

  The imaging unit 103 captures a still image or a moving image. In the present embodiment, an image of a subject that is affected by the light of the screen displayed on the display unit 102 is captured. It is attached to the “position where the image of the image can be taken”.

  The operation input unit 104 is a user interface such as a mouse or a keyboard, and is used for inputting various instructions to the calculation unit 101 by a user of the image processing apparatus 100. The display unit 102 and the operation input unit 104 may be integrated to form a touch panel screen. Further, the display unit 102 and the operation input unit 104 may be combined to form a touch panel screen and hard keys.

  The communication unit 105 is for performing data communication with an external device, and is a network controller or the like corresponding to LAN, 3G, 4G, Bluetooth (registered trademark), RFID (Radio Frequency IDentification), or the like.

  The measurement unit 106 is a combination of an illuminance sensor, a camera, a GPS (Global Positioning System) sensor, an electronic compass, a gyro sensor, and the like, and measures the relative position, direction, movement, illuminance, and the like of an external light source with respect to the image processing apparatus 100. .

  The RAM 108 has an area for storing computer programs and data loaded from the external storage unit 109 and computer programs and data received from the outside via the communication unit 105. Further, the RAM 108 has a work area for the arithmetic unit 101 to execute various processes. As an example of these areas, the “various work areas” in FIG. 1 can be used. Thus, the RAM 108 can provide various areas as appropriate.

  The external storage unit 109 is a nonvolatile memory such as a flash memory or a hard disk drive (HDD). The external storage unit 109 stores, for example, an OS (Operating System) and computer programs and data used in various processes described below. The computer program includes a boot program, a shooting program, a display control program, and a shot image correction program. Further, this data includes display information, color correction map data, gradation correction reference image, light source parameter, photographed image data, and various table data.

  Computer programs and data stored in the external storage unit 109 are appropriately loaded into the RAM 108 under the control of the calculation unit 101 and are processed by the calculation unit 101. The external storage unit 109 may be a device that reads / writes information from / to a storage medium such as an optical disk or a magnetic disk.

  The image processing apparatus 100 having the above configuration operates in response to various inputs from the operation input unit 104 and various inputs via a network supplied from the communication unit 105. That is, when an input from the operation input unit 104 or the communication unit 105 is supplied, an interrupt signal is first sent to the calculation unit 101. The arithmetic unit 101 issues various control signals by executing computer programs and data stored in the external storage unit 109, and various controls are performed in accordance with these control signals.

  Note that FIG. 1 only shows the main configuration required in the following description, and does not show all the configurations. In addition, as long as each processing described below can be realized, the configuration is not limited to the configuration illustrated in FIG. 1, and other configurations may be employed.

  Next, the user's (subject) viewing the display unit 102 in a dark place, the user's captured image captured by the imaging unit 103 is transmitted from the light from the display unit 102 or an external light source (external light source). A series of processing for correction in consideration of the light and the like will be described with reference to the flowchart of FIG.

  In the processing after step S401, the calculation unit 101 detects that the user of the apparatus operates the operation input unit 104 to input an imaging start instruction, or the communication unit 105 receives an imaging start instruction from the outside. This is what happens. However, unless the calculation unit 101 performs such detection, the calculation unit 101 is in a standby state. When such a detection is performed, the processing from step S401 is performed.

  The trigger for starting the processing after step S401 is not limited to these, and for example, an activation instruction for an analysis application such as health management or beauty management may be used as a trigger.

  In addition, in order to describe with specific examples below, the image processing apparatus 100 is a mobile device such as a smartphone or a tablet terminal, and the display unit 102 displays a display screen (display information) including images and characters. It is assumed that the photographing unit 103 is provided on the same surface. In the following, in a state where the user is viewing the display screen of the display unit 102, the imaging unit 103 captures the user, and the captured image obtained by the imaging is used as light from the display unit 102 or external Correct the light considering the light from the light source. Examples of such a system include a health management system and a beauty management system.

<Step S401>
The calculation unit 101 controls the shooting unit 103 by executing the shooting program, and the shooting unit 103 starts imaging under the control of the calculation unit 101. In order to collect a certain amount of captured images to be analyzed, it is assumed that imaging is performed regularly and automatically. However, imaging may be performed through a user operation and is suitable for analysis. Any form may be adopted as long as it can be collected. In any case, in step S401, the imaging unit 103 captures a plurality of (user's) captured images by continuously capturing images of the user.

  As described above, the imaging unit 103 is attached to “a position where an image of a user affected by the light of the screen displayed on the display unit 102 can be captured”. The image of the “user who is affected by the light on the screen” is captured. In the case of FIG. 2A, the photographing unit 103 captures an image of the user 202 receiving light from the display unit 102 displaying the display screen 203. 201 is an external light source.

<Step S402>
The calculation unit 101 acquires the measurement result obtained by the measurement unit 106. In this case, the measurement unit 106 includes a parameter (light source parameter) relating to the external light source 201 such as a relative position of the external light source 201 with respect to the image processing apparatus 100 and illuminance (environment light intensity). Measure such parameters.

<Step S403>
The calculation unit 101 refers to the light source parameter among the parameters acquired in step S402, and determines whether or not the “light intensity by the external light source 201” included in the light source parameter is equal to or greater than a predetermined intensity. That is, it is determined whether or not the captured image is captured with a light amount equal to or greater than a specified amount. If the result of this determination is that “the intensity of light from the external light source 201” is equal to or greater than the prescribed intensity, the external light source 201 is determined to be a non-weak light source and the process proceeds to step S407. On the other hand, if “the intensity of light from the external light source 201” is less than the prescribed intensity, the external light source 201 is determined to be a weak light source, and the process proceeds to step S404. As an example, when the illuminance obtained using the illuminance sensor included in the measurement unit 106 is lower than a threshold (200 lux or the like), the external light source 201 is determined to be a weak light source.

  Note that other methods may be adopted as a method of determining whether or not the external light source 201 is a weak light source. For example, the correlation between the flickering of the brightness of the display screen that appears in the captured image acquired by the image capturing in step S401 and the illuminance (luminance value, etc.) of the subject surface that appears in the captured image is quantified, You may employ | adopt the method of determining a threshold value.

<Step S404>
The calculation unit 101 obtains the relative position D and angle θ of the user 202 with respect to the image processing apparatus 100 using the captured image acquired by the image capturing in step S401. For example, the calculation unit 101 recognizes the position and orientation of the face of the user 202 included in the captured image acquired in step S401, and determines “the face of the user 202 with respect to the image processing apparatus 100 from the recognized face position and orientation”. Relative angle θ ”. In addition, the calculation unit 101 obtains the size of the face of the user 202 included in the captured image acquired in step S401, and calculates a distance value associated with the obtained size in advance as “the user for the image processing apparatus 100”. 202, relative position D ". Note that the method for obtaining the relative position and angle of the user 202 with respect to the image processing apparatus 100 is not limited to this. For example, when a target other than the face of the user 202 is targeted, a marker is given to the target in advance, and a relative position D and a relative angle θ are obtained based on the marker in the captured image, or relative by various sensors. The position D and the relative angle θ may be obtained.

<Step S405>
The calculation unit 101 acquires a screen shot of the display screen (copy image of the display screen) displayed on the display unit 102 at the time of image capturing in step S401, and the backlight brightness of the display unit 102 at the time of image capturing in step S401. To do. For this purpose, the calculation unit 101 refers to a table (display information table) illustrated in FIG. 3A stored in the external storage unit 109.

  Every time an image is captured, the calculation unit 101 stores the captured image and a screen shot of the screen displayed on the display unit 102 at the time of the image capture in the external storage unit 109, and the following information Register the group in the display information table. In each row of the display information table, an information group collected every time the imaging unit 101 captures an image is registered.

Display information ID: ID unique to the display screen displayed on the display unit 102 at the time of image capturing
Display date and time: Save date and time of screen shot of display screen displayed on display unit 102 at the time of image capture Screen shot ... File name of screen shot of display screen displayed on display unit 102 at the time of image capture (may be a path name)
Luminance: Brightness of backlight of display unit 102 at the time of image capture Average color: Average color of screen shot of display screen displayed on display unit 102 at the time of image capture Inclination ... Inclination direction of image processing apparatus 100 at the time of image capture ... Direction of the image processing apparatus 100 at the time of image capturing The inclination and direction of the image processing apparatus 100 at the time of image capturing are both measured using a gravity sensor, a decommissioning sensor, an electronic compass, and the like included in the measurement unit 106. It is a thing. It should be noted that such information registration in the display information table may be periodically performed regardless of image capturing when there is a surplus in the calculation capability of the calculation unit 101 and the capacity of the external storage unit 109. Absent.

  However, in step S405, the calculation unit 101 sets the value (file name) of “screen shot” registered in the row of the display date and time closest to the imaging date and time of image capturing in step S401 in the display information table of FIG. , To obtain the value of “luminance”. Then, the calculation unit 101 reads a screen shot corresponding to the acquired file name from the external storage unit 109 out of the screen shots recorded in the external storage unit 109 each time an image is captured (or periodically).

<Step S406>
The calculation unit 101 uses the relative angle θ and the relative position D obtained in step S404 to estimate the degree (influence level) of “the influence of light from the display screen displayed on the display unit 102” on the user's face. To do. For example, assuming that the backlight brightness acquired in step S405 is P, the degree of influence (P, θ, D) can be obtained by the following equation.

F (P, θ, D) = P × cos θ / D ²
Further, the calculation unit 101 analyzes the color characteristics of the screen shot. For example, the average luminance value (or intermediate value) of the R component of all the pixels of the screen shot, the average luminance value (or intermediate value) of the G component, and the average luminance value (or intermediate value) of the B component are used as the color characteristics of the screenshot. Collect as. That is, the information representing the deviation of the emission color in the display unit 102 at the time of image capturing in step S401 is collected.

  As shown on the left side of FIG. 2B, a display screen 206 having a color characteristic 207 in which the red component (R) and the blue component (B) of R, G, and B are relatively stronger than the green component (G) is displayed. It is assumed that it is displayed on the part 102. Light having the color characteristic 207 is emitted from the display unit 102 displaying the display screen 206 having such a color characteristic 207. Therefore, the color characteristic 209 of the captured image 208 captured by the imaging unit 103 of the user 202 viewing the display unit 102 displaying the display screen 206 having the color characteristic 207 in a dark place is the same as the color characteristic 207. Among R, G, and B, R and B have relatively stronger color characteristics than G.

  Further, as shown on the right side of FIG. 2B, it is assumed that a display screen 210 having a color characteristic 211 in which B of R, G, and B is relatively stronger than R and G is displayed on the display unit 102. Light having the color characteristic 211 is emitted from the display unit 102 that is displaying the display screen 210 having such a color characteristic 211. Therefore, the color characteristic 213 of the captured image 212 obtained by the imaging unit 103 capturing the user 202 viewing the display unit 102 displaying the display screen 210 having the color characteristic 211 in a dark place is the same as the color characteristic 211. Among R, G, and B, B has relatively stronger color characteristics than R and G.

  In this embodiment, since the influence according to the color characteristic of the display screen displayed on the display unit 102 when the captured image is captured is corrected, in this step, the display displayed on the display unit 102 when the captured image is captured. Get screen shot color characteristics.

<Step S407>
The calculation unit 101 acquires (generates) color correction map data A for performing color correction that cancels the influence of the color characteristics of the display screen acquired in step S406. For example, when the color characteristic acquired in step S406 is (R1, G1, B1), (1 / R1, 1 / G1, 1 / B1) is generated as the color correction map data A. For each pixel of the image to be corrected, this color correction map data A multiplies the R pixel value of the pixel by 1 / R1, multiplies the G pixel value of the pixel by 1 / G1, and This is for correcting the pixel value of the pixel by multiplying the pixel value by 1 / B1. Note that the generation method of the color correction map data A is not limited to a specific generation method, and a method for generating a color correction that cancels out “the influence of the color characteristic of the display screen acquired in step S406” that appears in the captured image is generated. If so, any generation method may be employed.

  In FIG. 2C, the influence of the color characteristic 207 on the color characteristic 209 of the captured image 208 of the user 202 viewing the display unit 102 displaying the display screen 206 having the color characteristic 207 in a dark place is offset. The color correction map data A214 for this is generated. By applying such color correction map data A214 to the captured image 208, a captured image 215 having a color characteristic 216 equivalent to that under a white light source can be obtained. In the case of FIG. 2C, the influence of light from the external light source 201 on the captured image is not considered. Therefore, in this step, the calculation unit 101 further acquires (generates) color correction map data B for correcting the influence of the light from the external light source on the captured image. As a method for generating the color correction map data B, for example, there is a method using a known technique using ambient light estimation or the like.

  Then, the calculation unit 101 synthesizes the color correction map data A and the color correction map data B at a ratio corresponding to the degree of influence F to generate combined color correction map data. For example, when the influence F is divided by 100 and normalized to a value of 0 to 1, it is generated as composite color correction map data = (1−F) × color correction map data A + F × color correction map data B. Can do. Such composite color correction map data is color correction map data for correcting a color deviation caused by the external light source 201 and the display screen 206 and correcting the color under the white light source, as shown in FIG. Hereinafter, the composite color correction map data may be simply referred to as color correction map data.

<Step S408>
The calculation unit 101 performs gradation correction map data for correcting the gradation characteristics of the corrected captured image obtained by performing color correction on the captured image obtained in step S401 using the composite color correction map data to reference gradation characteristics. (Tone correction information) is generated. If the subject can be clearly defined, a tone correction map may be created for each part so that the face area has a tone characteristic that matches between the plurality of captured images at the part such as the eyes, nose, and cheeks. Good.

  In the case of FIG. 2D, the gradation characteristic 217 of the corrected captured image 225 obtained by performing color correction on the captured image obtained in step S401 using the composite color correction map data is equivalent to the gradation correction reference image 218. Tone correction map data 219 for correcting the tone characteristics is generated. The captured image 225 is the same image as the captured image 215. The gradation correction reference image is an image that serves as a reference for gradation and hue under a white light source for an arbitrary subject, and uses an image taken outdoors. By correcting the gradation characteristic 217 of the captured image 225 using the gradation correction map data 219, the captured image 220 having the gradation characteristic 221 equivalent to that under the white light source is obtained.

  Note that the color and gradation correction target may be the entire image or the subject in the image. For the entire image, an appropriate gradation correction map is acquired from a fixed preset by light source estimation and subject detection. On the other hand, when the subject can be limited in advance, such as the face and body (health management, skin care management, etc.), the area is narrowed down using a face detection function or the like, and appropriate gradation correction is performed for each area.

  Note that the processing in steps S407 and S408 described above is performed when the processing proceeds from S403 → S404 → S405 → S406 → S407 → S408. When the process proceeds from S403 to S407 to S408, the conventional correction map data for color correction and correction map data for gradation correction are acquired in steps S407 and S408, respectively.

  Here, in order to perform the process of step S408 as described above, it is necessary to acquire a gradation correction reference image. Therefore, the arithmetic unit 101 refers to the table illustrated in FIG.

  The “corrected subject ID” is an ID unique to the subject included in the gradation correction reference image. “Creation date / time” is the creation date / time of the gradation correction reference image. The “gradation correction reference image file” is a file name (or a path name) of the gradation correction reference image. The “attribute label” is a label assigned to the subject included in the gradation correction reference image. However, in step S408, a “gradation correction reference image file name” in the same row as the correction subject ID same as the subject ID in the captured image acquired in step S401 is acquired, and the gradation correction reference of the acquired file name is acquired. Get an image.

<Step S409>
The gradation correction reference image used in step S408 is not always available. For example, a gradation correction reference image captured under a white light source and with sufficient illuminance (“the illuminance of the subject exceeds a threshold (200 lux, etc.)”, “the brightness value distribution non-whiteout / non-blackout area is 80% of the whole This corresponds to a case where there is no image that satisfies the criteria such as “the above”. Therefore, the calculation unit 101 determines whether the gradation correction map data has been successfully acquired because the gradation correction reference image has been acquired in step S409. As a result of this determination, when the gradation correction reference image has been acquired and the gradation correction map data has been successfully acquired, the process proceeds to step S411. On the other hand, if acquisition of the gradation correction map data fails because the gradation correction reference image cannot be acquired, the process proceeds to step S410.

<Step S410>
First, the calculation unit 101 acquires, from the external storage unit 109, a plurality of captured images captured in the past (including the same subject as the captured image acquired in step S401). The calculation unit 101 generates the color correction map data A for each acquired captured image as described above, and corrects the captured image using the generated color correction map data A. Then, the calculation unit 101 generates the corrected average image of each captured image (having the average pixel value of the pixel value of each captured image for each pixel) as the gradation correction reference image. In each of the corrected captured images, information loss such as blackout or whiteout occurs in each part of the subject due to insufficient illuminance. In order to compensate for these, such an average image is generated. Instead of the average image, an intermediate image (having the median pixel value of each captured image for each pixel) may be generated. Then, the calculation unit 101 generates gradation correction map data as described above using the generated gradation correction reference image.

<Step S411>
The calculation unit 101 generates a color-corrected captured image by performing color correction using the color correction map data obtained in step S407 on the captured image acquired in step S401. Further, the calculation unit 101 performs gradation correction using the gradation correction map data obtained in step S408 or step S410 on the color-corrected captured image, thereby generating a gradation-corrected captured image.

  Note that the arithmetic unit 101 creates and updates the tables shown in FIGS. 3C and 3D in the course of the above processing.

  In the table of FIG. 3C, “pre-correction image file name” is the file name of the captured image. “Shooting date / time” is the shooting date / time of the captured image. The “corresponding display information ID” is an ID unique to the display screen (screen shot) displayed on the display unit 102 when the captured image is captured. The “light source relative position” is a relative position of the external light source with respect to the image processing apparatus 100 measured in step S402. “Light source intensity” is the light intensity (illuminance) of the external light source measured in step S402. The “light source diffusion value” is the light diffusion value of the external light source measured in step S402. The “light source environmental value” is the environmental value of the light of the external light source measured in step S402.

  In the table of FIG. 3D, “pre-correction image file name” is the file name of the captured image. The “corrected subject ID” is an ID unique to the subject included in the captured image. “Image area coordinates” are the coordinates, width, and height of the center or four corners of the object area (rectangular area) in the captured image. “Rotation” is the rotation angle of the subject area with respect to the captured image. The “extracted file name” is a file name when separately storing a region where image correction has been performed on the captured image, and is a file name of a file to be stored for simple analysis processing such as health management.

  By the way, although the example in which the gradation correction reference image is sequentially updated in step S410 has been described above, when it is updated, the corrected image may be recorrected retroactively. In this way, all captured images can be corrected uniformly even for a subject that lacks the gradation correction reference image. In the case where the processing of correcting retroactively becomes a high load, the image processing apparatus 100 may be corrected together in a situation where the load is low.

  In addition to the change in the display unit, even when there is blinking of an external light source outside the angle of view, the above processing can be applied if the light source parameter corresponding to the captured image can be acquired appropriately. Furthermore, in addition to using a screen shot, the same effect can be obtained by using drawing hardware (display controller, GPU, etc.) or a drawing buffer of an operating system. By the way, there are cases where colors, gradation characteristics, and the like are different between the screen displayed on the display unit 102 and the light-emitting device serving as the display unit 102. In order to solve this problem, it is preferable to prepare a color / gradation profile and perform correction by converting the color and gradation. It should be noted that there are cases in which a desired gradation is not obtained due to occurrence of overexposure and blackout due to only correction using gradation correction map data. In this case, when shooting is performed so that a desired gradation is obtained by adding exposure conditions such as a camera aperture value, shutter speed, and ISO sensitivity to the photographing unit 103 in addition to the gradation correction map data. Good. In the above description, the captured image is an uncompressed image, but may be a compressed image such as a JPEG image if deterioration due to compression is allowed. In the above description, the number of display units included in the image processing apparatus 100 is one, but a plurality of display units may be provided. In this case, the color correction map data A may be generated using a combination of screen shot color characteristics in each display unit 102.

  As described above, according to the present embodiment, for example, even in shooting in a dark place or the like under a light emitting device such as a display whose display changes due to an operation by a person, the color and gradation are converted into a plurality of shot images. On the other hand, it can be corrected uniformly. Therefore, it is possible to improve the accuracy of analysis such as subject search using the corrected photographed image, health management, and beauty management. In the above example, an example of a device such as a smartphone in which a camera and a display are integrated has been described. However, the photographing unit and the display unit exist in the same system via a network line or the like, and interfere with each other in light emission and photographing. If it is a structure, it will not be limited to this. For example, when performing a person search using a subject image in shooting with a surveillance camera installed in the vicinity of digital signage, by applying this embodiment, a plurality of images of the same subject are all under a white light source. It is corrected considerably. Therefore, it is possible to accurately perform a search using clothes color as a search condition. In addition, a remote conference system using a projector and a camera in a low illuminance room, a communication system, and the like can be given. Since the configuration in which the person who sees the direction of the display that emits light is captured by the camera is the same as that of the smartphone, it can be applied in the same manner as the example of the present embodiment. By correcting the reproduced video to have a uniform color and gradation, there is an effect of improving visibility for the video viewer.

[Second Embodiment]
In the configuration of the image processing apparatus 100 according to the first embodiment, a configuration in which the photographing unit 103 and the image processing apparatus 100 are installed at locations separated from each other via a network is considered. In this case, depending on the status of the network path, various processing loads, etc., a case may occur in which it takes more time than usual for the captured image by the imaging unit 103 to be transferred to the image processing apparatus 100 and stored. sell. When such a case occurs, the captured image is registered in the image processing apparatus 100 as a captured image at an imaging time that is Δt (> 0) after the actual imaging time. However, the imaging time of the captured image of the subject affected by the light on the display screen displayed on the display unit 102 at time t is (t + Δt), and the captured image and the screen shot of the display screen are used. The generated correction map data is not appropriate. Therefore, in this embodiment, in such a case, when the captured image includes the display screen of the display unit 102, a screen shot of the display screen is extracted from the already recorded screen shot. Used to generate color correction map data.

  In the following, differences from the first embodiment will be described mainly, and unless otherwise noted, the same as the first embodiment.

  Next, the user's (subject) viewing the display unit 102 in a dark place, the user's captured image captured by the imaging unit 103 is transmitted from the light from the display unit 102 or an external light source (external light source). A series of processing for correction in consideration of the light and the like will be described with reference to the flowchart of FIG.

  Steps S501 to S504 and S509 to S513 are the same as steps S401 to S403, S405, and S407 to S411 in FIG.

<Step S505>
The calculation unit 101 identifies the subject that is shown in the captured image obtained in step S501.

<Step S506>
The calculation unit 101 determines whether a part or all of the display screen of the display unit 102 is included. For this determination, for example, a feature amount for recognizing a surface on which the screen is displayed on the display unit 102 such as a frame portion of the display surface of the display unit 102 is held in advance, and whether the feature amount exists in the captured image. This is done by judging whether or not. In addition, for example, a method of attaching a space marker to the display unit 102 for recognizing the display surface and recognizing the display surface, extracting a contour from a captured image, or dividing an area using a blinking change may be used. Of course, the technique for realizing the determination process in step S506 is not limited to this.

  As a result of the determination, if the captured image includes a part or all of the display screen of the display unit 102, the process proceeds to step S507. If not, the process proceeds to step S508. move on.

<Step S507>
The calculation unit 101 creates a screen shot that has been created in the past date and time and recorded in the external storage unit 109 from the shooting date and time of the captured image acquired in step S401 and has already been recorded in the external storage unit 109. Select from. For this specified amount, for example, a plurality of images are shot while causing a blinking change on the display screen, and the difference between the actual blinking change timing and the timing at which it is imaged is defined as the above prescribed amount. It is obtained and stored in the external storage unit 109.

<Step S508>
When the process proceeds from S506 to S507, the calculation unit 101 performs the processes of steps S404 and S406 using the screen shot selected in step S507. On the other hand, when the process proceeds from S506 to S508, the calculation unit 101 performs the processes of steps S404 and S406 using the screen shot selected in step S504. Thereby, the amplification (howling) of blinking can also be prevented. For example, in a conference system using a projector, the problem that the blinking conversion propagates to each other and the visibility is lowered can be solved.

  In the above example, timing correction was described, but in addition to this, a process for improving the accuracy of gradation correction is performed by measuring the illuminance change due to the luminance change of the display screen and the blinking change of the actual display unit. May be. Furthermore, when it is determined that the subject to be corrected is backlit as a result of acquiring the relative positions of the display unit, the subject, and the photographing unit, only color correction, tone correction, And apply exposure correction. By doing so, it is possible to obtain a photographed image obtained by photographing the correction target subject with an appropriate exposure.

[Third Embodiment]
When the time period in which the change in the time axis direction of the display screen (change in display content, brightness, brightness, etc.) is large is known in advance, or when it is determined that the amount of change is large, the interval between image capturing by the photographing unit 103 is shortened. Thus, the number of images taken per unit time may be increased. By using the captured image obtained in this way, captured images with various illuminance conditions from dark to bright can be obtained. Therefore, by combining them, a captured image with a wide dynamic range can be obtained. Specifically, after quantifying the time change of the screen shot of the display information, screen brightness, or relative position with the subject using a difference value or the like, the frequency of shooting is increased when it exceeds a threshold value. Good. And the process according to the flowchart of FIG.4 and FIG.5 is performed using the composite picked-up image obtained in this way.

  In the first embodiment, tone correction map data is estimated from a plurality of images in step S410. However, a change in the display screen may be induced so as to improve the accuracy of the tone correction reference image. Thereby, improvement in accuracy of the gradation correction reference image can be promoted. Specifically, a screen change is prompted so as to guide a display screen in which the illuminance increases when the blackout area is excessive in the gradation correction reference image, and a display screen in which the illuminance decreases when the overexposed area is excessive. Specifically, on a smartphone or the like, a recommendation notification is sent to the user to select and switch an application that can obtain an appropriate illuminance or user posture for acquiring the gradation correction reference image from a plurality of applications being executed. Do. By doing so, the brightness of the display screen changes, and a captured image of the subject suitable for obtaining the gradation correction reference image can be obtained. As a result, it is possible to improve the estimation accuracy of the gradation correction reference image. A part or all of the embodiments and modifications described above may be used in appropriate combination.

  Further, according to the above-described embodiment and modification, the user's captured image captured by the imaging unit in a state where the user is browsing the display unit in a dark place can be obtained from light from the display unit or an external light source. Can be corrected to an image equivalent to that under a white light source. Thereby, the accuracy of analysis such as subject search using the captured image, health management, and beauty management can be improved.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  101: Calculation unit 102: Display unit

Claims (9)

An image processing apparatus,
Acquisition means for acquiring a captured image of a subject;
Generating means for generating color correction information for performing color correction to cancel the color characteristics based on the color characteristics of the image displayed on the display unit at the time of capturing the captured image;
An image processing apparatus comprising: a correcting unit that corrects the captured image based on the color correction information. Furthermore,
Means for determining whether or not the captured image has been captured with a light amount equal to or greater than a specified amount;
The image processing apparatus according to claim 1, wherein the generation unit and the correction unit perform the generation and the correction, respectively, when the captured image is captured with a light amount less than a predetermined amount. The generating means further includes
The image processing apparatus according to claim 1, wherein the color correction information is combined with color correction information for correcting an influence of light from an external light source on the captured image. The generating means includes
The image processing apparatus according to claim 3, wherein the composition is performed at a ratio according to a relative position and a relative angle between the image processing apparatus and the subject. Furthermore,
Means for generating gradation correction information for correcting the gradation characteristics of the captured image to the gradation characteristics of a reference image that is an image serving as a reference for gradation and hue under a white light source;
The correction unit corrects the captured image based on the color correction information, and corrects the captured image based on the correction based on the gradation correction information. The image processing apparatus according to item. Furthermore,
For each of a plurality of past captured images, corresponding color correction information is generated, the captured image is corrected based on the color correction information generated for the captured image, and based on the corrected captured images. The image processing apparatus according to claim 5, further comprising means for generating the reference image.   7. The acquisition unit according to claim 1, wherein the acquisition unit is an imaging unit that images the subject, and is provided at a position where the subject browsing the screen of the display unit can be imaged. Image processing apparatus. An image processing method performed by an image processing apparatus,
An acquisition step in which an acquisition unit of the image processing apparatus acquires a captured image of a subject;
A generation step in which the generation unit of the image processing device generates color correction information for performing color correction that cancels the color characteristic based on the color characteristic of the image displayed on the display unit when the captured image is captured. When,
An image processing method comprising: a correction unit that corrects the captured image based on the color correction information.   The computer program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.

Images