JP2016080971A - Visual assistance system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To propose a useful visual assistance system.MEANS FOR SOLVING THE PROBLEM: A visual assistance system for correcting an image obtained by an imaging part which images an object that a user is trying to see in accordance with a visual defect of the user by an image processing part, and for displaying the image, and for selecting a plurality of stored processing modes is configured to restrict a manual operation unsuitable for selection on the basis of the detection of a situation sensor which detects a situation during use, and to automatically select the processing mode on the basis of the detection of the situation sensor, and to learn and store the detection of the situation sensor when the manual operation is performed, and to automatically select the processing mode on the basis of learning and storage and the detection of the situation sensor, and to determine the propriety of automatic selection on the basis of the learning and storage when the processing mode is automatically selected, and to decrease the frame rate of image display in image enlargement, and to stop the change of display when the situation sensor detects fine vibrations, and to restore the processing of the image processing part to a standard state on the basis of the detection of the situation sensor.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a visual aid system.

  Various visual assistance systems for visually impaired people have been studied. The main causes of visual impairment include glaucoma cases, cataracts, night blindness, eye diseases such as age-related macular degeneration, or developmental disorders such as visual impairment in early childhood. A device has been proposed. As an example, in an eyeglass-type visual enhancement device, image processing is performed on an image of a region corresponding to the user's visual field among images captured by a CCD camera, and this can be visually recognized by a user using a virtual image display device. Proposed. As another example, the image information obtained from the imaging unit is processed and displayed on the display area of the display unit, and the image of the outside world and the processed image of the display area are simultaneously displayed on the display unit by the observer. It has been proposed that can be seen from. (Patent Document 2)

JP 2003-287708 A Japanese Patent No. 4600200

  However, there are many issues to be further examined regarding the visual assistance system.

  In view of the above, an object of the present invention is to propose a more useful visual assistance system.

  In order to achieve the above object, the present invention provides an imaging unit that captures an object to be viewed by a user, an image processing unit that corrects an image obtained by the imaging unit according to a visual impairment of the user, and an image A display unit that displays an image of the processing unit so that a user can see it, a storage unit that stores a plurality of processing modes by the image processing unit, and a selection unit that selects from the plurality of processing modes. Provide a visual aid system. This makes it possible to select a processing mode suitable for the use situation.

  According to a specific feature of the present invention, manual operations unsuitable for selection are limited based on detection of a state sensor that detects a state of use. For example, the operation of selecting the image enlargement mode when the situation sensor detects acceleration is limited. Thereby, an accident etc. can be prevented. In the above, it is possible to notify the user that the manual operation is inappropriate. According to another feature, the processing mode of the storage unit is automatically selected based on the detection of the situation sensor. This enables automatic selection of a processing mode suitable for the usage situation.

  According to a specific feature of the present invention, the storage unit learns and stores a processing mode suitable for a situation based on detection of the situation sensor when a manual operation is performed. Accordingly, the processing mode of the storage unit can be automatically selected based on the learning storage and the detection of the situation sensor corresponding thereto. In addition, when automatically selecting the processing mode of the storage unit based on the detection of the situation sensor, it is possible to determine whether the automatic selection is appropriate based on the learning storage.

  According to another aspect of the present invention, an image capturing unit that captures an object to be viewed by a user, an image processing unit that corrects an image obtained by the image capturing unit in accordance with a visual impairment of the user, and image processing A visual aid comprising: a display unit for displaying a part of the image so that a user can see; and a control unit for reducing a frame rate of image display in the display unit when the image processing unit enlarges the image. A system is provided. Thereby, it is possible to alleviate video sickness and the like when the image is enlarged.

  According to another aspect of the present invention, an image capturing unit that captures an object to be viewed by a user, an image processing unit that corrects an image obtained by the image capturing unit in accordance with a visual impairment of the user, and image processing A display unit that displays the image of the unit so that the user can see, a status sensor that detects a situation during use, and a control unit that stops changing the display on the display unit when the status sensor detects minute vibrations A visual aid system is provided that comprises: Thereby, it is possible to alleviate video sickness and the like due to minute vibrations of the image.

  According to another aspect of the present invention, an image capturing unit that captures an object to be viewed by a user, an image processing unit that corrects an image obtained by the image capturing unit in accordance with a visual impairment of the user, and image processing A display unit that displays the image of the unit so that the user can see it, a status sensor that detects a situation during use, and a control unit that changes the processing of the image processing unit based on the detection of the status sensor A featured visual aid system is provided. Thereby, automatic display change according to the situation becomes possible.

  According to another aspect of the present invention, an image capturing unit that captures an object to be viewed by a user, an image processing unit that corrects an image obtained by the image capturing unit in accordance with a visual impairment of the user, and image processing A display unit that displays the image of a part so that the user can see it, a status sensor that detects a situation during use, a change unit that changes processing of the image processing unit, and the image processing unit based on detection of the status sensor And a control unit that returns the processing to the standard state. As a result, for example, when the situation sensor detects acceleration due to the start of walking, etc., the possibility of an accident is reduced by automatically returning from the image enlargement state for reading purposes to the same magnification state or the wide state for those with constricted visual field Can do.

  As described above, a more useful visual aid system is provided.

It is a block diagram which shows the whole structure in Example 1 of the visual assistance system of this invention. Example 1 3 is a basic flowchart illustrating an operation of a central control unit according to the first embodiment. It is a flowchart which shows the detail of step S22 of FIG. 2, and step S24. It is a flowchart which shows the detail of step S26 of FIG. 2, and step S28. It is a flowchart which shows the detail of step S34 of FIG. It is a flowchart which shows the detail of step S14 of FIG. It is a flowchart which shows the detail of the image sudden change mitigation process in Example 2 of the visual assistance system of this invention. (Example 2) It is a block diagram which shows the whole structure in Example 3 of the visual assistance system of this invention. (Example 3) 10 is a basic flowchart illustrating an operation of a central control unit according to a third embodiment. It is a flowchart which shows the detail of step S214 of FIG.

  FIG. 1 is a block diagram showing an overall configuration in Example 1 of a visual assistance system according to an embodiment of the present invention. The visual assistance system according to the first embodiment includes a goggle type head mounted display (hereinafter, “HMD”) 2 and a controller 4 connected to the goggle type head mounted display (hereinafter, “HMD”). The cable serves as a parallel data communication line and a power supply line for the HMD 2 and the controller 4.

  The HMD 2 is put in front of normal glasses 10 put in front of the user's right eye 6 and left eye 8. As described above, the HMD 2 is used on the assumption that the problem of refraction of the user's right eye 6 and left eye 8 is solved by the glasses 10. For this purpose, the HMD 2 includes a main body portion 2a and a temple portion 2b. When the temple portion 2b is put on the ear from above the glasses 10, the main body portion 2a is arranged in front of the lens of the eyeglasses 10.

  Both the right-eye display 12 and the left-eye display 14 of the HMD in the main body 2a are configured by an OLED (organic light emitting diode) display panel using an organic EL phenomenon. The drive unit 16 drives the right-eye display 12 and the left-eye display 14 based on image signals sent from the controller 4 as described later, and displays a right-eye image and a left-eye image. The virtual image of the displayed image is guided to the right eye 6 and the left eye 8 by the right eyepiece optical system 18 and the left eyepiece optical system 20 along the lines of sight 6a and 8a, as indicated by broken line arrows. The drive unit 16 also adjusts the focus of the right-eye eyepiece optical system 18 and the left-eye eyepiece optical system 20 based on the control of the controller 4, and moves the optical axis in parallel to shift the line of sight from the line of sight 6a or 8a. Shift adjustment is also performed.

  The right-eye image pickup element 22 in the main body 2a has a right-eye bending zoom lens optical system 24 that bends light incident along the line of sight 6a of the right eye 90 degrees inward (right side in the drawing) as indicated by a broken arrow. By this, a real image of the object scene is formed. Similarly, the left-eye image pickup device 26 is covered by a left-eye bending zoom lens optical system 28 that bends light incident along the line of sight 8a of the left eye 90 ° inward (left side in the drawing) as indicated by a dashed arrow. A real image of the scene is formed. The object scene images picked up by the right-eye image pickup device 22 and the left-eye image pickup device 26 are sent to the controller 4 via the drive unit 16 as described later. Due to the zooming function of the right-eye bent zoom lens optical system 24 and the left-eye bent zoom lens optical system 28, the right-eye image pickup device 22 and the left-eye image pickup device 26 not only have the same magnification image but also the actual image. It is possible to form an image in which the field is enlarged or an image in which the wide area of the actual scene is aggregated. The former is suitable for magnifying observation, and the latter is suitable for providing a field image in the field of view that is wider than it would actually look for a user with field stenosis.

  In the imaging system using the right-eye bent zoom lens optical system 24 and the left-eye bent zoom lens optical system 28, the thickness in the direction of the incident optical axis is reduced so that the main body 2a does not protrude excessively forward. Further, the right-eye bent zoom lens optical system 24 and the left-eye bent zoom lens optical system 28 occupy a space in the direction perpendicular to the lines of sight 6a and 8a, respectively, and the right-eye image further inside the bending direction. An imaging element 22 and a left-eye image imaging element 26 are disposed. This arrangement avoids the arrangement of parts that block the outside of the lines of sight 6a and 8a, so that the actual field outside the lines of sight 6a and 8a can be seen directly. Although it is said that the human eye can recognize information on a wide-angle field of about 200 degrees, the information on the field displayed on the right-eye display 12 and the left-eye display 14 in Example 1 is about 40. Degree. Therefore, in the first embodiment, visual information can be obtained by directly viewing the object scene outside the image information displayed on the right-eye display 12 and the left-eye display 14.

  With the above-described configuration, the visual assistance system according to the first embodiment sends the object scene image captured by the right-eye image imaging element 22 and the left-eye image imaging element 26 to the controller 4 for processing according to the user's symptoms. By displaying the processed image returned from the controller 4 on the right-eye display 12 and the left-eye display 14 so as to be observable, it is possible to obtain better visual information than when the object scene is viewed directly. For example, a user with night blindness (dark adaptation disorder) is provided with a processed image in which the gain is increased and the dark portion is raised by gamma correction. On the other hand, a processed image in which a high-intensity portion is compressed by gamma correction is provided to a user of dawn (light adaptation disorder). In order to improve the legibility of characters and the like, it is also possible to provide a black and white inverted image. Furthermore, visual information can be obtained by directly viewing the object scene around the image information displayed on the right-eye display 12 and the left-eye display 14 as described above. As will be described later, the direct image is harmonized with the display image by controlling the transmittance.

  As described above, in the first embodiment of the present invention, in the normal state, the optical axis of the right-eye eyepiece optical system 18 that guides the virtual image of the right-eye display 12 to the right eye 6 is incident on the right-eye bent-smooth lens optical system 24. It coincides with the optical axis. Similarly, the optical axis of the left-eye eyepiece optical system 20 that guides the virtual image of the left-eye display 14 to the left eye 8 coincides with the incident optical axis of the left-eye bent smooth lens optical system 28. If necessary, the optical axis of the right-eye eyepiece optical system 18 or the left-eye eyepiece optical system 20 can be moved in parallel by the control of the drive unit 16 so as to be shifted from the line of sight 6a or 8a. This is because, for example, when there is a disturbance in central vision due to age-related macular degeneration or the like, the image of the center of the field imaged by the right-eye image pickup device 22 or the left-eye image pickup device 26 is other than the central portion of the retina where there is no failure. This is to shift the line of sight so that it can be seen.

  Further, as described above, the first embodiment of the present invention is configured so that the actual field outside the lines of sight 6a and 8a can be directly seen as the background. A right-eye variable transmittance ND filter 33 is provided in the optical path from the outside of the right eye indicated by the arrow 30. The right-eye variable transmittance ND filter 33 is constituted by, for example, a liquid crystal shutter, and the transmittance is variable between the maximum transmittance and the light-shielded state under the control of the driving unit 16. Similarly, a left-eye variable transmittance ND filter 36 is provided in the optical path from the outside of the left eye indicated by the arrow 34, and the transmittance is controlled between the maximum transmittance and the light-shielded state under the control of the driving unit 16. It is variable. Thus, the transmittance of the right-eye variable transmittance ND filter 32 and the left-eye variable transmittance ND filter 36 can be changed independently of each other. The change in transmittance is used to assist the pupil's adaptability to changes in ambient brightness, and the display is made easier to see by changing the transmittance according to the change in brightness of the display. It is used to achieve harmony with direct observation images. Further, when black and white reverse display is performed on the display unit, the right-eye variable transmittance ND filter 32 and the left-eye variable transmittance ND filter 36 are set in a light-shielded state so as not to interfere with black-and-white reverse image observation.

  As is clear from FIG. 1, all the components in the first embodiment described above are housed in the main body 2a, and there is no portion protruding from the front surface of the main body 2a. Therefore, when viewed from a person facing the user wearing the HMD2, the HMD2 is felt to be similar to normal sunglasses, and the uncomfortable feeling that it is observed with a special device is reduced.

  The drive unit 16 in the main body 2a of the HMD 2 is connected to the controller 4 through parallel data communication and a power supply line 38, and performs mutual communication and power supply from the controller 4 to the HMD 2. In addition, in order to change the image processing provided to the right-eye display 12 and the left-eye display 14 according to the brightness of the environment, an ambient light sensor 40 is provided on the vine portion 2b of the HMD 2 and the communication line 42 is used to Light information is sent to the controller 4. Further, an acceleration sensor 44 is provided in the temple portion 2b to alleviate a sudden change in the image especially when the face is changed, and the information such as the movement of the face is sent to the controller 4 through the communication line 46. Yes. The parallel data communication and power supply line 38 and the communication lines 42 and 46 are actually combined into one connection cable. 1 shows a configuration in which the ambient light sensor 40, the acceleration sensor 44, and the biological sensor 45 directly communicate with the controller. However, the parallel light communication and the power supply line 38 communicate with each other via the drive unit 16. It may be configured.

  The controller 4 has an input / output unit 48 for communicating with the HMD 2 and supplying power to the HMD 2 as described above. The image processing unit 50 of the controller 4 processes the image received by the supply line 38 via the drive unit 16 from the image sensor 22 for the right eye image and the image sensor 26 for the left eye image of the parallel data communication and power supply HMD 2, and Is sent to the display control unit 52 as image data suitable for assistance. Image data from the display control unit 52 is transmitted by parallel data communication and the power supply line 38, and the drive unit 16 drives the right-eye display 12 and the left-eye display 14 based on the received image data to display an image. The background control unit 54 controls the eye variable transmittance ND filter 32 and the left eye variable transmittance ND filter 36 through parallel data communication and the power supply line 38.

  The preset storage unit 56 stores image processing information corresponding to individual symptoms of the user and ambient light, and preset values of transmittance information in the variable transmittance ND filter. The operation unit 58 performs the preset value input operation and the image processing selection operation such as black and white reversal in cooperation with the display of the controller display unit 60. The central control unit 62 controls the image processing unit 50 by adding the operation of the operation unit 58 and the information from the ambient light sensor 40 and the acceleration sensor 44 to the image processing information of the preset storage unit 56. Further, the central control unit 58 controls the background control unit 50 by adding information from the ambient light sensor 40 to the transmittance information in the preset storage unit 56. The control data of the background control unit 50 is transmitted by the parallel data communication and the power supply line 38, and based on the data received by the driving unit 16, the transmittances of the right-eye variable transmittance ND filter 32 and the left-eye variable transmittance ND filter 36 are set. Vary the background brightness directly observed. The central control unit 62 further controls the display control unit 52 and the controller display unit 60 in relation to the above functions. The power supply unit 64 supplies power to the entire controller and also supplies power to the HMD 2 via the input / output unit 48.

  FIG. 2 is a basic flowchart for explaining the operation of the central control unit 62 in the first embodiment. The flow starts when power supply to the system is started, and it is checked in step S2 whether a preset value is stored. If there is a memory, the preset value is read from the preset memory 56, and the process proceeds to step S6. On the other hand, if the preset value is not stored in step S2, the process proceeds to step S8, a default value indicating that no correction is performed in the image processing is read, and the process proceeds to step S6.

  In step S6, imaging by the right-eye image imaging device 22 and the left-eye image imaging device 26 is started, and the process proceeds to step S10, where the standard state display control based on the predetermined brightness and the standard state background corresponding thereto are performed. Then, transmittance control of the right-eye variable transmittance ND filter 32 and the left-eye variable transmittance ND filter 36 is started.

  Next, in step S12, it is checked whether or not an operation for setting a preset value has been performed. If it is confirmed that the setting operation has been performed, the process proceeds to step S14 to perform a preset value setting process, and the process proceeds to step S16. . On the other hand, when the preset value setting operation is not confirmed in step S12, the process directly proceeds to step S16. Details of the preset value setting process in step S14 will be described later.

  In step S16, it is checked whether or not the fact that the user has a central visual field defect is stored as a preset value. If so, the process proceeds to step S16 to perform a line-of-sight shift process, and the process proceeds to step S20. On the other hand, when it is confirmed in step S16 that the user does not correspond to the central visual field disorder, the process directly proceeds to step S20. At this time, the normal state is reached, and the optical axes of the right-eye eyepiece optical system 18 and the right-eye eyepiece optical system 20 are the same as the incident optical axes of the right-eye bent-summed lens optical system 24 and the left-eye bent-summed lens optical system 26, respectively. Will match.

  In step S16, it is checked whether or not the brightness of the ambient light has changed. If there is a change, the right-eye display change process in step S22 and the right-eye display change process in step S24 are sequentially executed to reach step S26. . In this way, the display change processing for the right eye and the left eye is performed independently. In step S26, the right eye background change process is performed, and then the left eye background change process of step S28 is executed, and the process proceeds to step S30. In this way, independent processing is performed for the right-eye and left-eye background change processing. On the other hand, when there is no change in ambient light in step S20, the process directly proceeds to step S30.

  In step S30, it is checked whether or not a black and white reversal operation has been performed. If there is an operation, the flow proceeds to step S32 to perform black and white reversal display and the background is shifted to step S34 in a light-shielding state so as not to prevent observation of black and white reversal display. . On the other hand, when the black and white reversal operation cannot be performed in step S30, the process directly proceeds to step S34. In step S34, it is checked whether the power is being supplied. If the power is being supplied, the process returns to step S12. Thereafter, as long as it is confirmed that the power is being supplied in step S36, steps S12 to S36 are repeated. On the other hand, if it is not confirmed in step S36 that power is being supplied, the flow is immediately terminated.

  FIG. 3 is a flowchart showing details of the right-eye display change process in step S22 and the left-eye display change process in step S24 of FIG. 2, and the contents are common to both steps, but for the right eye as shown in FIG. And for the left eye respectively. When the flow starts, the process proceeds to step S42, and it is checked whether or not the change in ambient light detected in step S20 in FIG. 1 is greater than or equal to a predetermined value that is predetermined for display change processing. If the change is equal to or less than a predetermined value that does not require display change processing, the flow is immediately terminated, and the process proceeds to step S26.

  On the other hand, when a change greater than or equal to the predetermined value is detected in step S42, the process proceeds to step S44, and it is checked whether or not the ambient light has increased beyond a predetermined value due to the change. If it is detected in step S44 that the ambient light has increased above the predetermined value, the gain of the image sensor is decreased in step S46, and the process proceeds to step S48. In step S48, it is checked whether or not the user has a light adaptation disorder. If yes, the process proceeds to step S50 to perform gamma correction for compressing the high-intensity part, and the process proceeds to step S52. In step S52, contour enhancement processing is further performed, and the process proceeds to step S54. On the other hand, if it is not detected in step S44 that the ambient light has increased above the predetermined value, or if it is not confirmed in step S48 that the user is a light adaptation disorder, the process proceeds directly to step S54.

  In step S54, it is checked whether ambient light has decreased below a predetermined value. When it is detected that the ambient light has decreased below the predetermined value, the gain of the image sensor is increased in step S56, and the process proceeds to step S58. In step S58, it is checked whether or not the user has a dark adaptation failure, and if applicable, the process proceeds to 60 to perform gamma correction for lifting the low luminance part, and the process proceeds to step S62. In step S62, contour enhancement processing is further performed, and the process proceeds to step S64. On the other hand, if it is not detected in step S54 that the ambient light has decreased below the predetermined value, or if it is not confirmed in step S58 that the user is a dark adaptation disorder, the process directly proceeds to step S64.

  In step S54, a counter for correcting display change according to the time when the pupil reacts to the change in brightness is reset and started, and the process proceeds to step S66. In step S66, it is checked whether or not the pupil response correction based on the previous brightness change is being performed. If yes, the process proceeds to step S68, the previous pupil response correction is canceled, and the process proceeds to step S70. On the other hand, if it is not detected in step S66 that the previous pupil reaction correction is being performed, the process proceeds directly to step S70. In step S70, pupil response correction is started, and processing for automatically ending pupil response correction is started when the pupil response is completed based on the counter, and the flow ends.

  FIG. 4 is a flowchart showing details of the right-eye background changing process in step S26 and the left-eye background changing process in step S28 of FIG. 2, and the contents are common to both steps, but for the right eye as shown in FIG. And for the left eye respectively. When the flow starts, the process proceeds to step S82, and it is checked whether the ambient light has increased beyond a predetermined value due to the change in ambient light detected in step S20 in FIG. Normally, the predetermined value in step S82 in FIG. 4 is lower in level than the predetermined value in step S44 in FIG.

  If it is detected in step S82 that the ambient light has increased above the predetermined value, the process proceeds to step S84, and the transmittance of the variable transmittance ND filter is decreased corresponding to the increase in ambient light, and the process proceeds to step S86. In step S86, it is checked whether or not the display portion change processing has been performed based on the current ambient light change. If yes, the process proceeds to step S88 to change the transmittance of the variable transmittance ND filter in response to the display portion change. Then, step S90 is reached. On the other hand, if it is not detected in step S82 that the ambient light has increased above the predetermined value, or if it is not confirmed in step S86 that the display unit changing process has been performed, the process proceeds directly to step S90.

  In step S90, it is checked whether or not the ambient light has decreased below a predetermined value due to the change in ambient light detected in step S20 of FIG. Usually, the predetermined value in step S90 of FIG. 4 has a higher level than the predetermined value in step S54 of FIG. When it is detected in step S90 that the ambient light has decreased below the predetermined value, the process proceeds to step S92, and it is checked whether or not the transmittance of the variable transmittance ND filter has already been maximized. If it is not the maximum value, the process proceeds to step S94, the transmittance of the variable transmittance ND filter is increased corresponding to the increase in the ambient light, and the process proceeds to step S96. However, this increase is limited by the maximum transmittance. On the other hand, if it is detected in step S92 that the transmittance of the variable transmittance ND filter has already been maximized, the process proceeds directly to step S96.

  In step S96, it is checked whether or not the display portion changing process has been performed based on the current ambient light change. If applicable, the process proceeds to step S98 to change the transmittance of the variable transmittance ND filter in response to the display portion change. Then, the process proceeds to step S100. However, this increase is limited by the maximum transmittance. On the other hand, if it is not detected in step S90 that the ambient light has decreased below the predetermined value, or if it is not confirmed in step S96 that the display unit changing process has been performed, the process proceeds directly to step S90.

  In step S100, it is checked whether or not there is a pupil response correction process started for display change in FIG. 3, and if applicable, the process proceeds to step S102 to start the transmittance correction of the corresponding variable transmittance ND filter. In response to the trend response correction for display change, a process for automatically ending the correction is started and the flow is ended.

  FIG. 5 is a flowchart showing details of the sudden image change mitigation process in step S34 of FIG. When the flow starts, it is checked in step S112 whether the display magnification is equal to or larger than the same magnification. If this is not the case, that is, if the display magnification is the same as or lower than the background magnification and the face orientation is not changed, etc. Then, the process proceeds to step S36.

  On the other hand, when it is detected in step S112 that the display magnification is equal to or larger than the same magnification, the process proceeds to step S114, and it is detected whether or not an acceleration based on the change of the face orientation is detected. When acceleration is detected, the process proceeds to step S116, the display of the previous frame on the right-eye display 12 and the left-eye display 14 is maintained, and the process proceeds to step S118. In step S118, it is checked whether or not a predetermined time corresponding to the display magnification (for example, a time corresponding to 3 frames at a display magnification of 2) has elapsed. If no time has elapsed, the process returns to step S116, and thereafter, step S116 and step S118 are repeated until the time elapse is detected in step S118, and the previous frame is maintained. On the other hand, when the passage of time is detected in step S118, the process proceeds to step S120.

  In step S120, acceleration detection is performed again. When acceleration is no longer detected due to the stop of the movement of the face, the process proceeds to step S122, the next frame of the frame maintained in step S116 is displayed, and the process proceeds to step S126. . The next frame display in step S122 is performed at a frame rate faster than usual. In step S126, it is checked whether or not the current frame has been displayed. If the current frame has not been caught, the process returns to step S120. Thereafter, unless a new acceleration is detected in step S120 and the current frame cannot be caught in step S126, steps S120 to S126 are repeated, and return to the current frame is performed at a frame rate faster than the normal frame rate. Then, when it is detected in step S126 that the current frame is displayed and the camera is in the saddle state, the flow is terminated. This alleviates sudden changes in the image when the face direction is changed in a state where the magnification is large, and delays the movement of the image. This delay is quickly recovered when the facial movement stops. On the other hand, when the acceleration is detected in step S120 and the facial motion continues, the process proceeds to step S124, the current frame is displayed, and the flow is ended. Therefore, when the movement of the face continues, the sudden change of the image is reduced in a form that is thinned out of the frame.

  FIG. 6 is a flowchart showing details of the preset value setting process in step S14 of FIG. When the flow starts, it is checked in step S132 whether the setting is made by a doctor. If it corresponds, it progresses to step S134, a doctor setting process is performed, and it transfers to step S136. On the other hand, if it is not detected in step S132 that the setting is made by the doctor, the process directly proceeds to step S136. In step S136, it is checked whether or not the setting is made by a vision trainer. If it corresponds, it progresses to step S138, a vision trainer setting process is performed, and it transfers to step S140. On the other hand, if it is not detected in step S136 that the setting is made by the vision trainer, the process proceeds directly to step S140.

  In step S140, it is checked whether the setting is made by the user himself / herself. If applicable, right eye setting is started in step S142, and ambient light initial setting is performed in step S144. In the personal setting, the user actually wears the HMD 2 and observes the right-eye display 12 to determine whether the setting is appropriate. Specifically, the display correction parameter is changed by the person in step S146. In step S148, the person's judgment is obtained as to whether or not the image observed by the right-eye display 12 is optimal. If it cannot be determined that it is optimal, the process returns to step S146, and thereafter, step S146 and step S148 are repeated to repeat the parameter change and the person's determination. Then, when the optimum determination is made by the user in step S148 and the operation unit 58 is operated, the process proceeds to step S150, the parameters in that state are stored, and the process proceeds to step S152.

  In step S152, it is checked whether the parameter storage stored as described above has reached a predetermined number of times. If the accumulated storage has not reached the predetermined number in Step S152, the process returns to Step S146, and Steps S146 to S152 are repeated until the accumulated storage reaches the predetermined number. On the other hand, when the accumulated storage reaches a predetermined number in step S152, the process proceeds to step S154, and the set parameter is determined by averaging the stored parameters.

  Next, in step S156, the ambient light is automatically changed for the purpose of setting, and the process proceeds to step S158. In step S158, it is checked whether or not the ambient light changing process has been completed. If the change process has not ended, the process returns to step S146, and thereafter, unless the ambient light change is ended, steps S146 to S158 are repeated, and the setting for the right eye is continued. On the other hand, when the ambient light changing process ends in step S158, the process proceeds to the left eye setting process in step S160. The details of the left-eye setting process in step S160 are the same as those in the right-eye setting process in steps S146 to S158, but are collectively shown in step S160 to avoid complication. When the left eye setting process in step S160 ends, the flow ends, and the process proceeds to step S16 in FIG. On the other hand, if it is not detected in step S140 that the setting is made by the user, the flow is immediately terminated.

  The implementation of the various features shown in the first embodiment is not limited to the first embodiment, and other embodiments can be implemented as long as the advantages can be enjoyed. For example, in the flow of FIG. 3, contour enhancement is performed when the user has a light adaptation disorder, and contrast enhancement is performed when the user has a dark adaptation disorder. It is possible to employ contour enhancement and contrast enhancement regardless of whether there is a light adaptation disorder or a dark adaptation disorder.

  FIG. 7 is a flowchart showing the details of the sudden image change mitigation process in Example 2 of the visual assistance system according to the embodiment of the present invention. The overall configuration of the second embodiment is the same as the block diagram of the first embodiment in FIG. The basic operation is the same as the basic flowchart of the first embodiment shown in the figure. Therefore, about a common part, Example 1 is used and description is abbreviate | omitted. The second embodiment differs from the first embodiment in the specific configuration of the image sudden change mitigation process in step S34 of FIG. The flowchart of FIG. 7 shows the details of step S34 of FIG.

  When the flow of FIG. 7 is started, whether or not an acceleration of a predetermined amount or more is detected is chucked in step S162. If acceleration is detected, the process proceeds to step S164, and it is checked whether or not a predetermined time (for example, 2 seconds) has elapsed since the acceleration was first detected in a series of acceleration detection. If the predetermined time has not elapsed, the process proceeds to step S166, a series of acceleration detection history analysis is performed, and the process proceeds to step S168. In step S168, whether or not a predetermined time (for example, 0.5 seconds) has elapsed since the analysis was started is checked. If the predetermined time has not elapsed, the process returns to step S166, and thereafter, the analysis is continued by repeating steps S166 and S168 until the predetermined time elapses.

  On the other hand, when it is confirmed that a predetermined time has elapsed since the analysis was started in step S168, the process proceeds to step S170, and whether or not a series of acceleration changes detected as a result of the history analysis corresponds to minute vibrations. Is checked. When the minute vibration is detected, the process proceeds to step S172, the display of the previous frame is maintained, and the display change is stopped. This is to prevent image blurring due to minute vibrations caused by unintentional body tremors. In step S174, it is checked whether or not acceleration in the same direction is detected.

  If the acceleration in the same direction is not detected in step S174, the process returns to step S164, and steps S164 to S174 are repeated unless it is detected in step S164 that a predetermined time has elapsed since the acceleration was first detected. On the other hand, when acceleration in the same direction is detected in step S174, it is determined that an intentional operation such as changing the orientation of the face has been performed, and the display is changed from the change stop state to the current frame. The process proceeds to S178. If it is determined in step S170 that it is not a minute vibration as a result of history analysis, the process immediately proceeds to step S178.

  In step S178, the current display magnification is confirmed, and the process proceeds to step S180 to check whether the display magnification is equal to or larger than the current magnification. If it is equal to or greater than the same magnification, in step S182, the magnification dependent frame dropping is instructed, and the process proceeds to step S184. In the magnification-dependent frame dropping in step S182, for example, the frame rate is reduced to half when the magnification is 1.5 times, and the frame rate is reduced to one third when the magnification is 2 times. The higher the magnification is, the lower the frame rate is so that the image does not move finely in a short time, thereby preventing video sickness caused by the enlarged image. On the other hand, if it is determined in step S180 that the image is not equal to or larger than the same size, the process proceeds to step S186 to instruct display at the normal frame rate, and the process proceeds to step S184.

  In step S184, the presence / absence of acceleration detection is detected again. If acceleration is subsequently detected, the process returns to step S164, and unless it is detected in step S164 that a predetermined time has elapsed since the acceleration was first detected, step S164 is performed. To repeat step S188. On the other hand, if it is confirmed in step S184 that no acceleration is detected, the process proceeds to step S188 to instruct the display of the current frame, and the flow ends.

  If it is detected in step S164 that a predetermined time has elapsed since the acceleration was first detected, the flow is immediately ended even if the acceleration detection is continued. This is to prevent other tasks in FIG. 2 from being unable to be executed due to staying in the flow of FIG. 7 for a long time. As apparent from FIG. 2, if there are no other tasks, the process from step S12 to step S32 is repeated. In step S34, the flow in FIG. 7 is repeated, and if there is acceleration detection, the corresponding function in FIG. 7 can be continued. If no acceleration is detected in step S162, the process proceeds to step S190, the display at the normal frame rate is instructed, and the flow ends. In this case, substantially no operation is performed in the flow of FIG. 7, but step S190 is set for the case where acceleration is not detected because the process reaches step S162 in a state other than the normal display state.

  FIG. 8 is a block diagram showing an overall configuration in Example 3 of the visual assistance system according to the embodiment of the present invention. Since the configuration of the third embodiment in FIG. 8 is similar to that of the first embodiment in FIG. 1, the same parts are denoted by the same reference numerals and the description thereof is omitted. The first point that the third embodiment differs from the first embodiment is that a pulse sensor 66 for detecting a pulse is provided in the temple portion 2b, and information such as whether the user is in a resting state or in an active state such as walking. Is sent to the controller 4 via the communication line 68. Similar to the first embodiment, the parallel data communication and power supply line 38 and the communication lines 42, 46, and 68 are actually combined into one connection cable. 3 also shows a configuration in which the ambient light sensor 40, the acceleration sensor 44, and the biological sensor 66 directly communicate with the controller 4. However, each sensor and the controller 4 perform parallel data communication via the drive unit 16. The power supply line 38 may be used for communication.

  The second point in which the third embodiment shown in FIG. 8 differs from the first embodiment shown in FIG. 1 is that a line-of-sight sensor is provided in the HMD 2 and detects the movement of the line of sight of the user. Information such as the movement of the user's line of sight detected by the line-of-sight sensor 70 is sent to the controller 4 via the drive unit 16 by parallel data communication. Details of the pulse sensor 66 and the line-of-sight sensor 70 will be described later. The ambient light sensor 40, the acceleration sensor 44, the pulse sensor 66, the line-of-sight sensor 70, and the like in the third embodiment are collectively referred to as a situation sensor because they detect the situation when the HMD 2 is used.

  The third point in which the third embodiment shown in FIG. 8 is different from the first embodiment shown in FIG. 1 is that the controller 4 has a mode storage unit 72, and each mode such as an enlarged mode, a wide mode, and a black / white reversal mode is used. It is a point that is registered together with a situation sensor for detecting a situation to be detected and learning information on the operation relationship of the operation unit 58. Then, in cooperation with the central control unit 74, the mode selection by the operation unit 58 is limited, or the registered mode is automatically selected. These details will also be described later.

  FIG. 9 is a basic flowchart for explaining the operation of the central control unit 74 in the third embodiment. Since the flow of FIG. 9 is often in common with the flow of the diagram in the third embodiment, common steps are denoted by the same step numbers and description thereof is omitted, and common step groups are also illustrated together. The description is omitted. That is, the start-up process in step S192 is a combination of steps S2, S4, and S8 in FIG. 2, step S194 is a combination of steps S12 to S18 in FIG. 2, and step S194 is a combination of steps S22 in FIG. Step S28 is summarized.

When the display / view change process in step S196 is followed to step S198, it is determined whether or not there has been a manual mode registration operation for registering an enlargement mode other than the normal mode, a wide mode, a black and white reversal mode, and the like. To check. When there is a mode registration operation, the process proceeds to step S200, a mode manual registration process according to the operation is executed, and the process proceeds to step S202. On the other hand, when the mode registration operation is not detected in step S198, the process directly proceeds to step S202.

  In step S202, it is checked whether or not a manual operation for selecting a mode has been performed. If there is a mode selection operation, the process proceeds to step S204 and the subsequent steps, and the state when the operation is performed is detected by each sensor. Specifically, ambient light is detected by the ambient light sensor 40 in step S204, acceleration is detected by the acceleration sensor 44 in step S206, and a pulse is detected by the pulse sensor 66 in step S208. In step S210, the line-of-sight sensor 70 detects the movement state of the line of sight.

  Next, in step S212, a mode automatic learning registration process is performed in which the detection state of each sensor is learned when the mode is manually selected, and the mode is automatically registered. In this process, for example, when the enlargement mode and the black and white mode are selected, if there is a rest state without acceleration detection and a rest state as viewed from the pulse, and the movement of the line of sight is limited, In the state, it is learned that the enlargement mode and the monochrome mode are selected, and these detection states are registered in the enlargement mode and the monochrome mode. In other words, in such a state, the user interprets and registers as having selected the enlargement mode and the monochrome mode for reading or viewing documents.

  The registration information by the mode automatic learning process in step S212 is used for cross-checking validity when automatically selecting a general mode as described later, and for establishing a specific learning result condition specific to the user. Used for automatic custom mode setting based. When the mode automatic learning process in step S212 ends, the process proceeds to step S214. On the other hand, if no mode selection operation is detected in step S202, the process directly proceeds to step S214. In step S214, mode change processing is executed, details of which will be described later. Step S34 and subsequent steps are the same as those in FIG.

  FIG. 10 is a flowchart showing details of the mode change process in step S214 of FIG. When the flow starts, it is checked in step S222 whether or not a manual operation for changing to the wide mode has been performed. If this operation is not detected, the process advances to step S224 to check whether or not a manual operation for changing to the enlargement mode has been performed. When a manual operation for changing to the enlargement mode is detected, the process proceeds to step S226, and it is checked whether or not an acceleration corresponding to the movement of the user is detected. If no acceleration is detected, the process proceeds to step S228, the magnification change according to the operation is performed (in this case, the magnification in the “enlargement mode”), and the process proceeds to step S230.

  On the other hand, when an acceleration corresponding to that the user is moving is detected in step S226, the process proceeds to step S232, and a notification display that enlargement is impossible is displayed on the controller display unit 60 to change the magnification. It transfers to step S230, without performing. Note that the message display indicating that enlargement is not possible may be superimposed on one of the images of the right-eye image sensor 22 and the left-eye image sensor 26.

  If it is detected in step S222 that a manual operation for changing to the wide mode has been performed, the process directly proceeds to step S228, and the magnification change according to the operation (in this case, the “wide mode” magnification) is executed. The process proceeds to step S230. In this way, if the operation is to reduce the magnification, even if the user is moving, there is little possibility of danger, so the acceleration is not detected and the operation is executed immediately. If the enlargement mode operation is not detected in step S224, it means that there is no manual operation for changing the magnification, and the process immediately proceeds to step S230.

  In step S230, it is confirmed whether or not the user is a visual field constrictor. If it is not a visual field constriction person, the process proceeds to step S236 to set the same size display to the standard mode and proceeds to step S236. On the other hand, if it is confirmed in step S230 that the user is a visual field constrictor, the process proceeds to step S238, the wide display is set to the standard mode, and the process proceeds to step S236.

  Steps 236 and thereafter relate to automatic mode change. First, in step S236, it is checked based on the acceleration sensor 44 whether or not the user is in a stationary state. If it is in a stationary state, the process proceeds to step S240, and it is checked based on the pulse sensor 66 whether the user is in a resting state. If it is in the stationary state, the process proceeds to step S242, and it is checked whether the automatic setting in the enlarged mode in such a state is consistent with the manual setting action of the user so far. If there is no contradiction as a result, the enlargement mode is automatically set in step S244, and the process proceeds to step S246. On the other hand, if the stationary state cannot be detected in step S236, the resting state cannot be detected in step S240, or inconsistency with the learning information is detected in step S242, the enlargement mode is not automatically set, and step S246 is performed. Migrate to The contradiction with the learning information means that, in the automatic registration process from step S202 to step S212 in FIG. 9, the history of manually selecting the expansion mode is rare despite being in a stationary state and a resting state, or This is the case where there is a history in which the enlargement mode automatically set based on the stationary state and the resting state is manually canceled. In such a case, there is a high possibility that the automatic setting of the enlargement mode is contrary to the user's intention even if it is detected that the camera is in a stationary state and a resting state. Will do.

  In step S246, the movement of the user's line of sight based on the output of the line-of-sight sensor 70 is compared with the reference data indicating the movement of the reading pattern, and it is checked whether the movement of the user's line of sight corresponds to the movement of the reading pattern. To do. If it is determined that the reading pattern line-of-sight movement is detected, the process proceeds to step S240, and whether the black-and-white reversal mode is automatically set in a state where the reading pattern line-of-sight movement exists is consistent with the manual setting action of the user so far. To check. If there is no contradiction as a result, the monochrome inversion mode is automatically set in step S250, and the process proceeds to step S252. On the other hand, when the reading pattern line-of-sight movement cannot be detected in step S246, the black-and-white reversal mode is not automatically set and the process proceeds to step S252. Note that the contradiction with the learning information in this case is the black and white reversal mode in the automatic registration process from step S202 to step S212 in FIG. Is a rare history, or a history in which the black-and-white reversal mode automatically set based on the reading pattern line-of-sight movement is manually canceled is recognized. In such a case, there is a high possibility that it is contrary to the user's intention to automatically set the black-and-white reversal mode even if the reading pattern line-of-sight movement is detected. Therefore, the process proceeds to step S254 with the automatic black-and-white reversal mode set ahead. become.

  In the automatic setting from step S236 to step S250, when the black / white reversal mode automatic setting in step S250 is performed after the enlargement mode automatic setting in step S244, the enlarged display is reversed in black and white. On the other hand, if the line-of-sight movement of the reading pattern is not detected in step S246 even if the enlargement mode is automatically set in step S244, there is a possibility that the user is sitting on a chair and performing assembly work or the like, which is suitable for character interpretation. Since it is often inappropriate to perform black and white reversal, only step S252 is reached by automatic setting of the enlargement mode.

  Further, the automatic setting in steps S236 to S250 described above relates to automatic setting of relatively general modes such as the enlargement mode and the black and white reversal mode. On the other hand, in steps S252 and S254, a plurality of modes that are custom-set for a specific user according to the user's symptoms such as an adaptation failure to light and dark, an abnormality in visual field, etc. (at least a specific mode under a specific condition) And a standard mode individually set for the user) are selected according to the situation and are automatically set. Specifically, it is checked whether or not the predetermined condition custom-set in step S252 is met, and if so, the automatic custom mode setting process in step S254 is entered to automatically change the mode setting. The process proceeds to S256. On the other hand, if it is not detected in step S252 that the custom setting condition is met, the process directly proceeds to step S256, and the current setting mode is maintained.

  In step S256, it is checked whether or not a movement acceleration is detected. When the movement acceleration is detected, the process proceeds to step S258, where the automatic return to the standard mode is performed and the flow ends. On the other hand, if the movement acceleration is not detected in step S256, the current setting mode is maintained and the flow ends. If the movement acceleration is detected in step S256, for example, a person sitting in a chair and reading or working may have stood up and started walking. In such a case, the enlargement mode and the black / white inversion mode are maintained. If it is done, it is dangerous, and in step S258, automatic return to the standard mode is performed. This standard mode is the same magnification mode or the wide mode set in step S234 or step S238. The standard mode can be custom set for a specific user.

  The implementation of the various features shown in the above embodiments is not limited to each embodiment, and other embodiments can be implemented as long as the advantages can be enjoyed. Further, in this specification, for the sake of simplicity, the explanations and illustrations in the respective embodiments are concentrated on portions different from those in the other embodiments. However, the embodiments having the respective characteristics described in the respective embodiments are replaced. Needless to say, the present invention can be implemented by combining a plurality of features or a plurality of features.

  The present invention can be applied to a visual assistance system.

22, 24, 26, 28 Imaging unit 50 Image processing unit 12, 18, 14, 20 Display unit 72, Storage unit 58, 74 for storing processing mode Selection unit 40, 44, 66, 70 Status sensor 74 Image display frame Control unit 74 for reducing rate Control unit 74 for stopping display change Control unit 74 for changing processing of image processing unit Control unit for returning processing of image processing unit to standard state

Claims (19)

  An imaging unit that captures an object to be viewed by the user, an image processing unit that corrects an image obtained by the imaging unit according to the visual impairment of the user, and an image displayed by the image processing unit so that the user can see the image A visual assistance system comprising: a display unit that performs storage; a storage unit that stores a plurality of processing modes by the image processing unit; and a selection unit that performs selection from the plurality of processing modes.   The visual assistance system according to claim 1, further comprising a situation sensor that detects a situation during use.   The visual assistance system according to claim 2, wherein the selection unit performs selection from the plurality of processing modes based on a manual operation.   4. The visual system according to claim 3, wherein the situation sensor detects a situation when a manual operation is performed, and the selection unit restricts a manual operation unsuitable for selection based on detection by the situation sensor. Auxiliary system.   The visual assistance system according to claim 4, wherein the selection unit notifies that selection is inappropriate.   The visual assistance system according to claim 4, wherein the imaging selection unit adds a restriction to an operation of selecting an image enlargement mode when the situation sensor detects acceleration.   The visual assistance system according to claim 2, wherein the storage unit learns and stores a processing mode suitable for a situation based on detection of the situation sensor when a manual operation is performed.   The visual assistance system according to claim 7, wherein the selection unit automatically selects a processing mode of the storage unit based on the learning storage of the storage unit and detection of the situation sensor corresponding to the learning storage.   8. The selection unit according to claim 7, wherein the selection unit automatically selects a processing mode of the storage unit based on detection of the situation sensor and determines whether or not the automatic selection is appropriate based on the learning storage of the storage unit. Visual aid system.   The visual assistance system according to claim 2, wherein the selection unit automatically selects a processing mode of the storage unit based on detection of the situation sensor.   The storage unit stores a standard mode, and the selection unit returns to the selection of the standard mode no matter what processing mode is selected based on detection of the status sensor. The visual assistance system according to claim 10.   The visual assistance system according to claim 11, wherein the standard mode is an equal magnification display mode.   12. The visual assistance system according to claim 11, wherein the standard mode is a wide display mode.   The visual assistance system according to claim 1, wherein when the image processing unit enlarges an image, a frame rate of image display on the display unit is reduced.   14. The visual assistance system according to claim 2, wherein when the status sensor detects minute vibrations, the display change in the display unit is stopped.   An imaging unit that captures an object to be viewed by the user, an image processing unit that corrects an image obtained by the imaging unit according to the visual impairment of the user, and an image displayed by the image processing unit so that the user can see the image And a control unit that reduces a frame rate of image display on the display unit when the image processing unit enlarges the image.   An imaging unit that captures an object to be viewed by the user, an image processing unit that corrects an image obtained by the imaging unit according to the visual impairment of the user, and an image displayed by the image processing unit so that the user can see the image A visual assistance system, comprising: a display unit that detects a situation during use; and a control unit that stops changing the display on the display unit when the situation sensor detects minute vibrations.   An imaging unit that captures an object to be viewed by the user, an image processing unit that corrects an image obtained by the imaging unit according to the visual impairment of the user, and an image displayed by the image processing unit so that the user can see the image A visual assistance system comprising: a display unit that detects a situation at the time of use; and a control unit that changes processing of the image processing unit based on detection of the situation sensor.   An imaging unit that captures an object to be viewed by the user, an image processing unit that corrects an image obtained by the imaging unit according to the visual impairment of the user, and an image displayed by the image processing unit so that the user can see the image A display unit that detects a situation during use, a change unit that changes processing of the image processing unit, and a control unit that returns the processing of the image processing unit to a standard state based on the detection of the status sensor A visual assistance system characterized by comprising:

Images