CN114863093B - Neural network training method based on eye movement technology and building design method and system - Google Patents
Neural network training method based on eye movement technology and building design method and system Download PDFInfo
- Publication number
- CN114863093B CN114863093B CN202210603543.7A CN202210603543A CN114863093B CN 114863093 B CN114863093 B CN 114863093B CN 202210603543 A CN202210603543 A CN 202210603543A CN 114863093 B CN114863093 B CN 114863093B
- Authority
- CN
- China
- Prior art keywords
- building
- data
- neural network
- design
- eye movement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 122
- 238000013461 design Methods 0.000 title claims abstract description 122
- 230000004424 eye movement Effects 0.000 title claims abstract description 99
- 238000012549 training Methods 0.000 title claims abstract description 94
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000005516 engineering process Methods 0.000 title claims abstract description 20
- 230000000007 visual effect Effects 0.000 claims abstract description 53
- 238000001514 detection method Methods 0.000 claims description 32
- 238000012360 testing method Methods 0.000 claims description 22
- 238000010586 diagram Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000000605 extraction Methods 0.000 claims description 14
- 230000000694 effects Effects 0.000 abstract description 4
- 210000001508 eye Anatomy 0.000 description 14
- 230000007246 mechanism Effects 0.000 description 11
- 238000010276 construction Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 210000004087 cornea Anatomy 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 4
- 230000008447 perception Effects 0.000 description 4
- 210000001747 pupil Anatomy 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003945 visual behavior Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000001028 reflection method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a neural network training method based on an eye movement technology and a building design method and a system, the method takes eye movement data under building elevation images and corresponding different demographic information as training data, the first neural network obtained through training can be used for predicting attention hot spots, visual focuses and eye movement tracks of people under different demographic information on the building elevation images, the data can be used for building design assistance, the effect of attention center assistance positioning is achieved, so that the building designer can conduct finer design on areas with attention, the second neural network is trained through a second training data packet, the second neural network can be deeply applied to building elevation image design with a white area, humanized and flexible assistance is provided for building design, the method is reliable in implementation, the neural network training data sources are wide, and the method has good popularization prospect in building design assistance after training to a converged model.
Description
Technical Field
The invention relates to the fields of vision technology, building design and neural network aided design, in particular to a neural network training method based on eye movement technology, and a building design method and system.
Background
With the widespread use of neural network (AI) technology, in the field of building design, some aided design software has introduced AI algorithms to provide reference advice services for designers, which are mainly based on AI to automatically generate building arrangements. Because architects often need to follow industry knowledge to design when performing building arrangements; the design is provided with a set of logic rule support, so that an AI technology is introduced, the set of logic rules is learned, and the model can be used for providing suggestions for automatically generating building arrangement schemes after training to convergence; specifically, most of the algorithmic training of AI neural networks in the past maps information from the real world into quantized data, finds the links between them, summarizes and applies rules, and thus forms neural networks for different scenarios based on training data.
Currently, under the age of artificial intelligence and big data, there is an increasing report of literature on the generation of new designs by a computer fitting design rules by building a machine learning model and then applying the trained model to the new design. Although computers are capable of learning and analyzing a large number of building drawings and also consider economic, scientific, comfort and other metrics, there is always a lack of perceptual understanding of building design, mainly one less perception of human perception. As architectural design practices move toward humanization and refinement, architects and planners are urgently required more means and methods to learn how people perceive the environment, which in turn affects people, thereby guiding the architect's design. As the perception of a person is mainly reflected in the visual behaviour of the person. Thus, performing an eye movement experiment can accurately record subtle eye movement behaviors of a human. The human visual behavior is researched by combining the eye movement technology, the perception degree of people on places is represented, and then through combination with a new algorithm in the field of data analysis, the interrelationship of environmental elements in the built environment can be deeply analyzed, so that new possibilities are brought to the aspects of design development, scheme evaluation and the like of urban design.
If the eye movement technology can be introduced into the neural network training of the building auxiliary design, the accuracy and the reliability of the neural network are realized by integrating all data of different demographic information, the assistance applied to the building design auxiliary is obvious, and positive practical significance is provided for the AI intervention building design.
Disclosure of Invention
In view of the above, the invention aims to provide a neural network training method based on eye movement technology, a building design method and a system, which are reliable in implementation, flexible in operation, high in response efficiency and humanized.
In order to achieve the technical purpose, the invention adopts the following technical scheme:
A neural network training method based on eye movement technology, comprising:
s01, inputting a building elevation drawing, and setting the building elevation drawing as first training data;
s02, displaying the first training data in the sight of a tester, recording the eye movement condition of the tester when watching the first training data, and generating eye movement data;
S03, acquiring eye movement data, generating attention hot spot data, visual focus data and eye movement track data according to the eye movement data, and setting the attention hot spot data, the visual focus data and the eye movement track data in an associated mode as analysis data;
S04, acquiring demographic information of a tester corresponding to the analysis data, and associating the analysis data, the first training data and the demographic information of the tester to generate a first training data packet;
S05, acquiring a first training data packet, inputting the first training data packet into a neural network for training, and acquiring a trained neural network;
s06, inputting the test data into the trained neural network to obtain an output result, converging the model when the output result meets the preset condition, and completing training of the neural network to obtain the first neural network.
As a possible implementation manner, in the solution S03, eye movement data is analyzed by BeGaze analysis software to generate attention hot spot data, visual focus data, and eye movement track data.
As a possible implementation manner, further, in the scenario S04, the demographic information includes one or more of age, education background, occupation, and ethnicity of the tester.
As a possible implementation manner, further, in the aspect S06, the test data is a test data packet extracted from or separately established from a plurality of first training data packets, where the test data packet has a building elevation view, demographic information data, and analysis data corresponding to the demographic information data one by one, the test data has the building elevation view, the demographic information data as input items, and the analysis data as reference output items;
after input to the trained neural network, the input term obtains an output result, which matches the reference output term,
When the matching value accords with the preset value, the model converges, the neural network training is completed, and a first neural network is obtained;
and when the matching value does not accord with the preset value, returning to S05.
For the acquisition aspect of eye movement data, the scheme is based on the eye movement data acquisition equipment in the prior art, and the general working principle, the operation flow and the functional introduction of the eye movement data acquisition equipment are briefly as follows:
Eye movement experiments, eye movement data acquisition was performed using a germany SMI eye movement instrument. The data acquisition is divided into eight steps: 1. the eye movement instrument is connected with the recorder 2, the recorder is started 3, the new experimental task 4, the conventional parameters 5 are set, the demographic information is input 6, the equipment is correctly worn 7, the three-point calibration is performed 8, and the data are acquired. After the eight steps, eye movement information of a tested person is acquired. Since the ages, educational backgrounds, etc. of the subjects are all different, we need to collect different subjects during the experiment.
After the eye movement data is collected, the eye movement data analysis is required to be performed by using Begaze analysis software. Thus, we can obtain a series of analysis charts, such as: visual focus map, attention heat map, eye movement track map, etc. In eye movement application studies, there are several commonly used indicators: 1. the visual index, namely the analysis chart 2, and the statistical analysis index, comprises basic indexes (gazing, eye jump and the like) and synthetic indexes.
① Principle and mechanism of visual attention: when the human brain and the visual nervous system perform visual processing on scene or image information, not all information is equally treated, but is used to allocate more visual attention to certain areas or targets. The visual attention mechanism is to mimic the way a human observes. In general, when a person looks at a picture, the person looks more attention to some local information of the picture than to grasp the picture from the whole. The limited visual processing capability is focused on the region of interest, improving the viewing efficiency.
From a physiological point of view, the ability of a person to process information is limited. The human eye covers approximately 120 degrees in the horizontal direction within the field of view. But only 2 degrees, the image outside the foveal area will be blurred, belonging to a clear foveal field. That is, our line of sight tends to select some objects while ignoring others, which is visual attention.
Visual attention is not just a physiological concept, but human vision is often related to what is considered that is of interest to the heart. Direct vision to a particular individual or location in the scene, with the exception of individuals or locations, a process known as visual attention mechanisms. The human visual attention mechanism comprises two basic mechanisms, bottom-up and top-down. The bottom-up attention mechanism is driven by external stimuli and features, responsible for rapid, automatic and non-autonomous rapid transition of attention and gaze. The top-down mechanism is task driven, based on empirical memory, everyone is different, person-to-person. Thus, the information selection strategy of the human visual system guides people to notice significant areas in massive data and allocate resources to process important information by using a visual attention mechanism.
At this time, through eye movement experiments, the visual attention of people is finally calculated by using an accurate and scientific method, so that people can know how to find the region of interest and what determines the attention of people, and how to perceive the external environment is explained. These help us further elucidate the relationship between visual quality improvement and spatial distribution.
② Principle of eye movement instrument: human eye movement is very subtle, and we need to collect eye movement data by means of a certain scientific instrument, namely an eye movement instrument. The eye tracker has three components, the first being a scene camera, which is located in the middle of two frames, recording the experimental scene based on the perspective of the subject. The second is a near infrared light source that emits light while producing reflection at our eyes. The third is an eye movement sensor which records the reflection of the retina and cornea, calculates the position of the gaze, and then superimposes it on the video taken by the scene camera. Through these three components, the infrared emitter emits infrared rays into our eyes, at this time, the light reflected by our cornea is unchanged, but the light reflected by our pupil is changed, so the gazing position of our subject is recorded by the change of the angle between the light reflected by cornea and the light reflected by pupil.
Specifically, the pupil-cornea reflection method is to take an eye image with an eye camera, and then obtain a pupil center position by image processing. Then, the cornea reflection point (yellow spot) is taken as a base point of the relative positions of the eye camera and the eyeball as shown in fig. 2, and the sight line vector coordinate can be obtained according to the pupil center obtained by image processing, so that the eye point of regard of the human eye is determined. On the basis, a mapping function between a vector formed by pupils and cornea reflection points and a screen fixation point is found out through a plurality of calibration programs, and then the interest point of a person at the position of fixation in a screen is tracked in real time by detecting the variation of the pupils-cornea vector, so that an eye movement track and an interest result are obtained.
③ Three-point scaling in data acquisition: taking SMIETG eye tracker as an example, 3-point calibration is a process of matching the gaze point acquired by ETG with the actual gaze point of the subject. 3-point scaling requires 3 scaling points, which 3 scaling points have to form the shape of a triangle and cannot be on the same straight line. Meanwhile, the testee is told about the accurate positions of 3 calibration points. Next, let the testee watch first calibration point, click the screen, move the cross sign (eye point that the eye tracker gathered) on the screen to the calibration point (the position that the testee actually watched), finally, accomplish 3 point calibration in proper order.
④ Two deep learning techniques:
1. image study based on attention model
Visual attention mechanisms refer to the automatic processing of regions of interest by humans in the face of a scene, and the selective omission of regions of no interest, which are referred to as saliency regions. In the field of computer vision, the research related to "attention" is roughly divided into two directions, namely, saliency detection for the purpose of "purely seeking salience" and a visual attention model (also called a focusing model) taking a "attention mechanism for other matters" as thought, wherein the two models take the simulation of human eye attention as core research contents, and different attention needs to be paid to different positions in an input scene in order to enable the model to realize targeted "focusing". Wherein the visual attention model is a core module of the attention mechanism as a model for locating regions of significant differences between different objects. For example, a test chart (see fig. 1) is given with its left chart as the original chart, and the attention-focusing area shown in the right chart can be predicted by image study based on the attention model.
Based on the above scheme, the invention also provides a building design method, which comprises the neural network training method based on the eye movement technology, and comprises the following steps:
a01, acquiring a building elevation view to be processed and demographic information, and generating data to be processed;
a02, inputting data to be processed into a first neural network for eye movement data prediction, and obtaining a prediction result;
A03, according to a prediction result, obtaining attention hot spot data, visual focus data and eye movement track data of the building elevation map to be processed under corresponding demographic information;
a04, outputting building design auxiliary information according to the attention hot spot data, the visual focus data and the eye movement track data.
As a possible implementation manner, the solution further includes:
b01, building a building design database, a design learning database and a demographic information database, wherein building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
b02, importing the building elevation map and preset demographic information in the design learning database into a first neural network to conduct eye movement data prediction, and obtaining a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data; then, according to the attention hot spot data, the visual focus data and the eye movement track data, corresponding areas in the building elevation view are positioned, and image features of a preset area range are extracted;
B03, importing the extracted image features into a detection neural network to identify buildings and building styles in the images, obtaining building detection results, then matching the building detection results with visual focus data to obtain focus buildings and supporting buildings, and associating the focus buildings and the supporting buildings;
B04, acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
b05, acquiring a second training data packet, inputting the second training data packet into the neural network for training, and acquiring a trained neural network;
And B06, inputting the test data into the trained neural network to obtain an output result, converging the model when the output result meets a preset condition, completing training of the neural network, and obtaining a second neural network, wherein the second neural network is used for outputting focal building suggestion information according to the focal building or outputting the focal building suggestion information according to the focal building.
As a possible implementation manner, the solution further includes:
b07, importing a building design elevation view with a blank area and marking the area to generate a building design elevation view to be processed and a region to be processed on the building design elevation view to be processed;
B08, extracting the region in the preset adjacent range of the region to be processed on the elevation view of the building design to be processed, then importing the extraction result into a detection neural network, and outputting building information by the detection neural network;
And B09, importing the building information and the preset demographic information into a second neural network, acquiring data output by the second neural network, and setting the data as suggested building information of the area to be processed.
As a possible implementation manner, further, the suggested building information described in the present embodiment B09 is focal building suggested information or setting-off building suggested information;
in addition, the building design database stores the focal building proposal information or the building information pointed in the setting-off building proposal information.
According to the technical scheme, the automatic design method for the building elevation based on the combination of the eye movement technology and the AI is provided, visual attention hot spots of users in the environment of the building elevation are identified through the eye movement technology, attention hot spot diagrams of different users are analyzed, and the deep learning model is utilized to extract characteristics and summarize rules. The automatic design system for the final building elevation is more effective by combining the existing AI technology, and meets the requirements of actual building design better.
Based on the above scheme, the invention also provides a building design system, which comprises:
The database unit is used for constructing a building design database, a design learning database and a demographic information database, building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
A first neural network unit for performing eye movement data prediction on the building elevation map and preset demographic information in the imported design learning database to obtain a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data;
The feature extraction unit is used for positioning corresponding areas in the building elevation according to the attention hot spot data, the visual focus data and the eye movement track data output by the first neural network unit, extracting image features of a preset area range, and extracting areas in a preset adjacent range of the area to be processed on the building design elevation to be processed;
The data scheduling unit is used for importing the building elevation images and preset demographic information in the design learning database into the first neural network and importing the image features extracted by the feature extraction unit into the detection neural network to identify the building and the building style in the image;
The detection neural network unit is used for identifying the buildings and the building styles in the images according to the image features extracted by the feature extraction unit, obtaining building detection results, detecting the areas in the preset adjacent ranges of the areas to be processed on the elevation view of the building design to be processed, and outputting building information;
The data association unit is used for matching the building detection result output by the detection neural network unit with the vision focus data to obtain a focus building and a setting-off building, and associating the focus building and the setting-off building; the system is also used for acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
The second neural network unit is used for training by the second training data packet and outputting suggested building information of the to-be-processed area on the to-be-processed building design elevation according to the building information and the preset demographic information.
Based on the above scheme, the invention also provides a computer readable storage medium, wherein at least one instruction, at least one section of program, code set or instruction set is stored in the storage medium, and the at least one instruction, the at least one section of program, the code set or the instruction set is loaded and executed by a processor to realize the building design method.
By adopting the technical scheme, compared with the prior art, the invention has the beneficial effects that: the building elevation image and eye movement data corresponding to different demographic information are used as training data, the training data is imported into the neural network to be trained, a first neural network is obtained, the first neural network can be used for predicting attention hot spot data, visual focus data and eye movement track data of the building elevation image by people under different demographic information, and accordingly the first neural network can be applied to building design assistance, so that when building designers design buildings aiming at different demographic information people or styles, the effect of attention center auxiliary positioning is achieved, the building designers can conduct more refined design on the area focused by people, the auxiliary positioning on partial data in a second training data packet can be achieved by means of the first neural network, after feature extraction and feature detection, a second training data packet can be obtained, the second neural network obtained through training of the second training data packet can be used for assisting building design, in addition, the building or supporting building suggestion pushing is carried out on a building elevation image white area of the designer, in addition, the second neural network can be applied to the building design assistance model, the second neural network is further applied to the building elevation image, the second neural network is applied to the building elevation image, the training model is provided, and the training model is widely applied to the building elevation image, and the training model is achieved, and the user has the advantages of being applied to the training model, and the user has the advantages of being widely, and the training model is achieved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of the eye movement technique mentioned in this scheme to extract attention hot spot data;
FIG. 2 is a schematic flow chart of a neural network training method based on eye movement technology according to the embodiment of the invention;
FIG. 3 is a schematic diagram of an embodiment of the present invention for collecting and importing test person's eye movement data into neural network training by eye movement technique;
FIG. 4 is a schematic flow chart of a schematic implementation of the building design method according to the embodiment of the present invention for outputting building design auxiliary information through a first neural network;
FIG. 5 is a schematic flow chart of a schematic implementation of a blank section of a guiding building design elevation after combining a first neural network and a second neural network according to the building design method of the embodiment of the invention;
Fig. 6 is a schematic block diagram of a building design system according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is specifically noted that the following examples are only for illustrating the present invention, but do not limit the scope of the present invention. Likewise, the following examples are only some, but not all, of the examples of the present invention, and all other examples, which a person of ordinary skill in the art would obtain without making any inventive effort, are within the scope of the present invention.
As shown in fig. 2, the neural network training method based on the eye movement technology according to this embodiment includes:
s01, inputting a building elevation drawing, and setting the building elevation drawing as first training data;
s02, displaying the first training data in the sight of a tester, recording the eye movement condition of the tester when watching the first training data, and generating eye movement data;
S03, acquiring eye movement data, generating attention hot spot data, visual focus data and eye movement track data according to the eye movement data, and setting the attention hot spot data, the visual focus data and the eye movement track data in an associated mode as analysis data;
S04, acquiring demographic information of a tester corresponding to the analysis data, and associating the analysis data, the first training data and the demographic information of the tester to generate a first training data packet;
S05, acquiring a first training data packet, inputting the first training data packet into a neural network for training, and acquiring a trained neural network;
s06, inputting the test data into the trained neural network to obtain an output result, converging the model when the output result meets the preset condition, and completing training of the neural network to obtain the first neural network.
The first neural network obtained through training can be used for carrying out visual attention prediction on the building elevation, namely, predicting possible attention focus, eye movement track and attention hot spot data of people with different demographic information on the building elevation, and in this way, the effect of attention center auxiliary positioning is achieved when building designers design buildings aiming at different demographic information people or styles, so that the building designers can design the areas with attention to people more finely.
Referring to fig. 3, in the aspect of collecting eye movement data, in the present embodiment S03, eye movement data is analyzed by BeGaze analysis software to generate attention hot spot data, visual focus data, and eye movement track data.
For the demographic information mentioned in this scenario, in this scenario S04, the demographic information includes one or more of age, educational background, occupation, ethnicity of the tester.
Because the reliability of the test data determines the verification reliability of the trained neural network model, in order to facilitate data extraction, in the scheme S06, the test data is extracted from a plurality of first training data packets, the test data packets are provided with building elevation diagrams, demographic information data and analysis data corresponding to the demographic information data one by one, the test data takes the building elevation diagrams and the demographic information data as input items, and the analysis data is taken as a reference output item;
after input to the trained neural network, the input term obtains an output result, which matches the reference output term,
When the matching value accords with the preset value, the model converges, the neural network training is completed, and a first neural network is obtained;
and when the matching value does not accord with the preset value, returning to S05.
The test data of the present solution is not limited to be extracted from the first training data packet, but may be a separately established test data packet.
With further reference to fig. 4, based on the above scheme, the present embodiment further provides a building design method, which includes the neural network training method based on the eye movement technology, including:
a01, acquiring a building elevation view to be processed and demographic information, and generating data to be processed;
a02, inputting data to be processed into a first neural network for eye movement data prediction, and obtaining a prediction result;
A03, according to a prediction result, obtaining attention hot spot data, visual focus data and eye movement track data of the building elevation map to be processed under corresponding demographic information;
a04, outputting building design auxiliary information according to the attention hot spot data, the visual focus data and the eye movement track data.
The building design auxiliary information output through the scheme is beneficial to playing the effect of focusing on the auxiliary positioning of the center of gravity when a designer performs building design on different demographic information personnel or styles, so that the building designer can design the region focused on by people more finely.
In addition to the above application, as shown in connection with fig. 5, the present embodiment further includes:
b01, building a building design database, a design learning database and a demographic information database, wherein building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
b02, importing the building elevation map and preset demographic information in the design learning database into a first neural network to conduct eye movement data prediction, and obtaining a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data; then, according to the attention hot spot data, the visual focus data and the eye movement track data, corresponding areas in the building elevation view are positioned, and image features of a preset area range are extracted;
B03, importing the extracted image features into a detection neural network to identify buildings and building styles in the images, obtaining building detection results, then matching the building detection results with visual focus data to obtain focus buildings and supporting buildings, and associating the focus buildings and the supporting buildings;
B04, acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
b05, acquiring a second training data packet, inputting the second training data packet into the neural network for training, and acquiring a trained neural network;
B06, inputting the test data into a trained neural network to obtain an output result, converging the model when the output result meets a preset condition, completing training of the neural network, and obtaining a second neural network, wherein the second neural network is used for outputting focal building suggestion information according to the set-off building or outputting the set-off building suggestion information according to the focal building;
b07, importing a building design elevation view with a blank area and marking the area to generate a building design elevation view to be processed and a region to be processed on the building design elevation view to be processed;
B08, extracting the region in the preset adjacent range of the region to be processed on the elevation view of the building design to be processed, then importing the extraction result into a detection neural network, and outputting building information by the detection neural network;
And B09, importing the building information and the preset demographic information into a second neural network, acquiring data output by the second neural network, and setting the data as suggested building information of the area to be processed.
In this embodiment, the construction information data such as the recommended construction information, the focus construction information, and the setting-off construction information may be stored in advance in the database, and then encoded for easy extraction.
The recommended construction information described in the present embodiment B09 is focal construction recommended information or setting off construction recommended information; in addition, the building design database stores the focal building proposal information or the building information pointed in the setting-off building proposal information.
As shown in fig. 6, based on the above-described scheme, the present embodiment further provides a building design system, which includes:
The database unit is used for constructing a building design database, a design learning database and a demographic information database, building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
A first neural network unit for performing eye movement data prediction on the building elevation map and preset demographic information in the imported design learning database to obtain a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data;
The feature extraction unit is used for positioning corresponding areas in the building elevation according to the attention hot spot data, the visual focus data and the eye movement track data output by the first neural network unit, extracting image features of a preset area range, and extracting areas in a preset adjacent range of the area to be processed on the building design elevation to be processed;
The data scheduling unit is used for importing the building elevation images and preset demographic information in the design learning database into the first neural network and importing the image features extracted by the feature extraction unit into the detection neural network to identify the building and the building style in the image;
The detection neural network unit is used for identifying the buildings and the building styles in the images according to the image features extracted by the feature extraction unit, obtaining building detection results, detecting the areas in the preset adjacent ranges of the areas to be processed on the elevation view of the building design to be processed, and outputting building information;
The data association unit is used for matching the building detection result output by the detection neural network unit with the vision focus data to obtain a focus building and a setting-off building, and associating the focus building and the setting-off building; the system is also used for acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
The second neural network unit is used for training by the second training data packet and outputting suggested building information of the to-be-processed area on the to-be-processed building design elevation according to the building information and the preset demographic information.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing description is only a partial embodiment of the present invention, and is not intended to limit the scope of the present invention, and all equivalent devices or equivalent processes using the descriptions and the drawings of the present invention or directly or indirectly applied to other related technical fields are included in the scope of the present invention.
Claims (9)
1. A neural network training method based on eye movement technology, comprising:
s01, inputting a building elevation drawing, and setting the building elevation drawing as first training data;
s02, displaying the first training data in the sight of a tester, recording the eye movement condition of the tester when watching the first training data, and generating eye movement data;
S03, acquiring eye movement data, generating attention hot spot data, visual focus data and eye movement track data according to the eye movement data, and setting the attention hot spot data, the visual focus data and the eye movement track data in an associated mode as analysis data;
S04, acquiring demographic information of a tester corresponding to the analysis data, and associating the analysis data, the first training data and the demographic information of the tester to generate a first training data packet;
S05, acquiring a first training data packet, inputting the first training data packet into a neural network for training, and acquiring a trained neural network;
s06, inputting test data into the trained neural network to obtain an output result, converging a model when the output result meets a preset condition, completing training of the neural network, and obtaining a first neural network;
In S06, the test data is a test data packet extracted from or separately built in a plurality of first training data packets, where the test data packet includes a building elevation, demographic information data, and analysis data corresponding to the demographic information data one by one, and the test data uses the building elevation and the demographic information data as input items, and the analysis data as reference output items;
after input to the trained neural network, the input term obtains an output result, which matches the reference output term,
When the matching value accords with the preset value, the model converges, the neural network training is completed, and a first neural network is obtained;
and when the matching value does not accord with the preset value, returning to S05.
2. The eye movement technique-based neural network training method of claim 1, wherein in S03, eye movement data is analyzed by BeGaze analysis software to generate attention hot spot data, visual focus data, and eye movement trajectory data.
3. The eye movement technique based neural network training method of claim 1, wherein in S04, the demographic information includes one or more of age, educational background, occupation, ethnicity of the tester.
4. A building design method comprising the eye movement technology-based neural network training method according to any one of claims 1 to 3, comprising:
a01, acquiring a building elevation view to be processed and demographic information, and generating data to be processed;
a02, inputting data to be processed into a first neural network for eye movement data prediction, and obtaining a prediction result;
A03, according to a prediction result, obtaining attention hot spot data, visual focus data and eye movement track data of the building elevation map to be processed under corresponding demographic information;
a04, outputting building design auxiliary information according to the attention hot spot data, the visual focus data and the eye movement track data.
5. A method of architectural design according to claim 4, further comprising:
b01, building a building design database, a design learning database and a demographic information database, wherein building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
b02, importing the building elevation map and preset demographic information in the design learning database into a first neural network to conduct eye movement data prediction, and obtaining a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data; then, according to the attention hot spot data, the visual focus data and the eye movement track data, corresponding areas in the building elevation view are positioned, and image features of a preset area range are extracted;
B03, importing the extracted image features into a detection neural network to identify buildings and building styles in the images, obtaining building detection results, then matching the building detection results with visual focus data to obtain focus buildings and supporting buildings, and associating the focus buildings and the supporting buildings;
B04, acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
b05, acquiring a second training data packet, inputting the second training data packet into the neural network for training, and acquiring a trained neural network;
And B06, inputting the test data into the trained neural network to obtain an output result, converging the model when the output result meets a preset condition, completing training of the neural network, and obtaining a second neural network, wherein the second neural network is used for outputting focal building suggestion information according to the focal building or outputting the focal building suggestion information according to the focal building.
6. A method of architectural design according to claim 5, further comprising:
b07, importing a building design elevation view with a blank area and marking the area to generate a building design elevation view to be processed and a region to be processed on the building design elevation view to be processed;
B08, extracting the region in the preset adjacent range of the region to be processed on the elevation view of the building design to be processed, then importing the extraction result into a detection neural network, and outputting building information by the detection neural network;
And B09, importing the building information and the preset demographic information into a second neural network, acquiring data output by the second neural network, and setting the data as suggested building information of the area to be processed.
7. The building design method according to claim 6, wherein the recommended building information in B09 is focus building recommended information or setting off building recommended information;
in addition, the building design database stores the focal building proposal information or the building information pointed in the setting-off building proposal information.
8. A building design system loaded with the building design method according to one of claims 4 to 7, the system comprising:
The database unit is used for constructing a building design database, a design learning database and a demographic information database, building design diagrams with different styles and specifications are stored in the building design database, a plurality of building elevation diagrams formed by various building arrangements are stored in the design learning database, and a plurality of pieces of demographic information are stored in the demographic information database;
A first neural network unit for performing eye movement data prediction on the building elevation map and preset demographic information in the imported design learning database to obtain a prediction result, wherein the prediction result comprises attention hot spot data, visual focus data and eye movement track data;
The feature extraction unit is used for positioning corresponding areas in the building elevation according to the attention hot spot data, the visual focus data and the eye movement track data output by the first neural network unit, extracting image features of a preset area range, and extracting areas in a preset adjacent range of the area to be processed on the building design elevation to be processed;
The data scheduling unit is used for importing the building elevation images and preset demographic information in the design learning database into the first neural network and importing the image features extracted by the feature extraction unit into the detection neural network to identify the building and the building style in the image;
The detection neural network unit is used for identifying the buildings and the building styles in the images according to the image features extracted by the feature extraction unit, obtaining building detection results, detecting the areas in the preset adjacent ranges of the areas to be processed on the elevation view of the building design to be processed, and outputting building information;
The data association unit is used for matching the building detection result output by the detection neural network unit with the vision focus data to obtain a focus building and a setting-off building, and associating the focus building and the setting-off building; the system is also used for acquiring the specifications of the focus building and the setting-off building, respectively associating the specifications to generate focus building data and setting-off building data, and then associating the focus building data, the setting-off building data and corresponding demographic information to generate a second training data packet;
The second neural network unit is used for training by the second training data packet and outputting suggested building information of the to-be-processed area on the to-be-processed building design elevation according to the building information and the preset demographic information.
9. A computer-readable storage medium, characterized by: the storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded by a processor and executed to implement the architectural design method according to one of claims 4 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210603543.7A CN114863093B (en) | 2022-05-30 | 2022-05-30 | Neural network training method based on eye movement technology and building design method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210603543.7A CN114863093B (en) | 2022-05-30 | 2022-05-30 | Neural network training method based on eye movement technology and building design method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114863093A CN114863093A (en) | 2022-08-05 |
| CN114863093B true CN114863093B (en) | 2024-05-31 |
Family
ID=82640608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210603543.7A Active CN114863093B (en) | 2022-05-30 | 2022-05-30 | Neural network training method based on eye movement technology and building design method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114863093B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9775512B1 (en) * | 2014-03-19 | 2017-10-03 | Christopher W. Tyler | Binocular eye tracking from video frame sequences |
| CN110070624A (en) * | 2019-04-26 | 2019-07-30 | 厦门大学 | A kind of cityscape feature knowledge method for distinguishing based on VR combination eye-tracking |
| CN110460837A (en) * | 2018-05-07 | 2019-11-15 | 苹果公司 | With central fovea display and the electronic equipment for watching prediction attentively |
| CN110737339A (en) * | 2019-10-28 | 2020-01-31 | 福州大学 | Visual-tactile interaction model construction method based on deep learning |
| CN111209811A (en) * | 2019-12-26 | 2020-05-29 | 的卢技术有限公司 | Method and system for detecting eyeball attention position in real time |
| CN111631736A (en) * | 2020-06-03 | 2020-09-08 | 安徽建筑大学 | Building indoor learning efficiency detection method and system |
| WO2020186883A1 (en) * | 2019-03-18 | 2020-09-24 | 北京市商汤科技开发有限公司 | Methods, devices and apparatuses for gaze area detection and neural network training |
| CN112890815A (en) * | 2019-12-04 | 2021-06-04 | 中国科学院深圳先进技术研究院 | Autism auxiliary evaluation system and method based on deep learning |
| CN113610145A (en) * | 2021-08-03 | 2021-11-05 | 上海联影智能医疗科技有限公司 | Model training method, image prediction method, training system, and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112987064B (en) * | 2021-02-09 | 2022-07-15 | 北京百度网讯科技有限公司 | Building positioning method, device, equipment, storage medium and terminal equipment |
-
2022
- 2022-05-30 CN CN202210603543.7A patent/CN114863093B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9775512B1 (en) * | 2014-03-19 | 2017-10-03 | Christopher W. Tyler | Binocular eye tracking from video frame sequences |
| CN110460837A (en) * | 2018-05-07 | 2019-11-15 | 苹果公司 | With central fovea display and the electronic equipment for watching prediction attentively |
| WO2020186883A1 (en) * | 2019-03-18 | 2020-09-24 | 北京市商汤科技开发有限公司 | Methods, devices and apparatuses for gaze area detection and neural network training |
| CN110070624A (en) * | 2019-04-26 | 2019-07-30 | 厦门大学 | A kind of cityscape feature knowledge method for distinguishing based on VR combination eye-tracking |
| CN110737339A (en) * | 2019-10-28 | 2020-01-31 | 福州大学 | Visual-tactile interaction model construction method based on deep learning |
| CN112890815A (en) * | 2019-12-04 | 2021-06-04 | 中国科学院深圳先进技术研究院 | Autism auxiliary evaluation system and method based on deep learning |
| CN111209811A (en) * | 2019-12-26 | 2020-05-29 | 的卢技术有限公司 | Method and system for detecting eyeball attention position in real time |
| CN111631736A (en) * | 2020-06-03 | 2020-09-08 | 安徽建筑大学 | Building indoor learning efficiency detection method and system |
| CN113610145A (en) * | 2021-08-03 | 2021-11-05 | 上海联影智能医疗科技有限公司 | Model training method, image prediction method, training system, and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 一种整合语义对象特征的视觉注意力模型;李娜;赵歆波;;哈尔滨工业大学学报;20200510(05);全文 * |
| 基于眼动实验的嘉庚风格建筑研究 ——以厦门大学建南大会堂为例;邱鲤鲤等;中国房地产（下旬刊）;20211110;73-79 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114863093A (en) | 2022-08-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11366517B2 (en) | Human-computer interface using high-speed and accurate tracking of user interactions | |
| CN108427503B (en) | Human eye tracking method and human eye tracking device | |
| Chen et al. | A probabilistic approach to online eye gaze tracking without explicit personal calibration | |
| Berton et al. | Eye-gaze activity in crowds: impact of virtual reality and density | |
| Sundstedt et al. | Visual attention and gaze behavior in games: An object-based approach | |
| Wang et al. | Automated student engagement monitoring and evaluation during learning in the wild | |
| Panetta et al. | Software architecture for automating cognitive science eye-tracking data analysis and object annotation | |
| CN109620266A (en) | The detection method and system of individual anxiety level | |
| Li et al. | Image understanding from experts' eyes by modeling perceptual skill of diagnostic reasoning processes | |
| Słowiński et al. | Assessment of cognitive biases in augmented reality: Beyond eye tracking | |
| CN117576771A (en) | Visual attention assessment method, device, medium and equipment | |
| Hutt et al. | Evaluating calibration-free webcam-based eye tracking for gaze-based user modeling | |
| Jeelani et al. | Real-world mapping of gaze fixations using instance segmentation for road construction safety applications | |
| CN114863093B (en) | Neural network training method based on eye movement technology and building design method and system | |
| Zhang et al. | Eye gaze estimation and its applications | |
| Lochbihler et al. | Assessing Driver Gaze Location in a Dynamic Vehicle Environment | |
| Nadella | Eyes detection and tracking and eye gaze estimation | |
| Gutstein et al. | Optical flow, positioning, and eye coordination: automating the annotation of physician-patient interactions | |
| Vranceanu et al. | A computer vision approach for the eye accesing cue model used in neuro-linguistic programming | |
| Bennett et al. | Looking at faces: autonomous perspective invariant facial gaze analysis | |
| Ferhat et al. | Eye-tracking with webcam-based setups: Implementation of a real-time system and an analysis of factors affecting performance | |
| Akshay et al. | E-commerce Website User Engagement Analysis: An Eye Tracking Viewpoint | |
| Qiao et al. | A Review of Attention Detection in Online Learning | |
| Zhang et al. | Region of interest detection based on visual perception model | |
| Patil et al. | Eye Gaze-Based Student Readability Analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |