Can We Trust Regression…
I would like to give a background as to what prompted this article. A few years back, word on the street was that the Collateral Underwriter was going to be using big data collected directly from our reports from the inception of UAD to aid in determining what should be “reasonable adjustments”. In order to be preemptive, I consulted an acquaintance of mine who just so happened to have a PhD in Statistics from Princeton. We worked through a number of real case scenario data sets to try and figure out if there was a way I could use regression to aid in determining adjustments. After hours of “playing” with these data sets, his opinion was adamant that an appraiser should not rely upon regression using MLS data or any other collected form of data to determine what an adjustment should be. It is unreliable and the analysis is static. Real estate is not static. He also pointed out that if you are using regression and don’t understand the outputs, R-values, P-values, coefficients, reliability factors, etc. then he would be able to rip someone to shreds on the stand who opted for any type of regression program that claims to be able to summarize adjustments based on regression.
Big data is the buzz word of the real estate industry right now. Multi-million dollar companies are popping into existence claiming to have the “right formula” for residential valuations – only to a few years later go bankrupt, (like Zaio which claimed to have the special sauce, only to re-brand as Clarocity which claimed the same, only to re-brand back to Zaio when their stock declined 98.5%, Or HouseCanary, or others). Fannie and Freddie claim to have the special sauce in the “Collateral Underwriter” but appraisers nationwide report that the output in all but the most uniform of areas is still just short of gibberish.
At the core of all of these algorithms is math, and much like stock market prediction, the math is complex, unproven and not for the faint of heart. Dr. Jason Osborne of NCSU gives 4 fundamental assumptions that must be true for multiple regression (the system at the core of these systems and most available to appraisers) to be reliable (read his paper here). These four assumptions are:
Homoscedasticity and Variables are normally distributed
This very large word means that the distribution falls evenly around the regression line. These both have to be tested on a case by case basis. However, since appraisers receive no mandatory college level statistical analysis, it’s too easy for appraisers to trust the tools that they are given that claim to be doing the analysis for them.
Variables are measured without error
At this point, most appraisers are laughing. Appraisers know that the data present in their local MLS has often been “fluffed,” (the word used in the real estate industry for what agents do to make a property look more appealing without outright lying). However, “fluffing” a 2 bedroom home with a windowless den in the basement into a 3 bedroom home is misleading at best. We also know that assessors are not always the most reliable home measures, sometimes including the below grade square footage with the above grade. The data sources that the regression here relies upon cannot be pushed through without significant cleaning. This is why multiple companies over the last 10 years have been stealing this data from each other, because the raw data is worthless.
A linear relationship between independent and dependent variables
On this point, decades of real estate education again teaches us that regression in real estate fails this test. The “Law of Diminishing Returns,” bluntly states that the relationship of amenities to value is NOT linear, but rather a diminishing curve. Land is the easiest example to showcase because vacant land sales prove it time and time again.
From here we see that the price per acre (vertical) of land decreases as the number of acres (horizontal) increases. This is the “Law of Diminishing Returns” at work. This is true of all of the amenities in real estate (Square footage, bathrooms, pools, etc). As the number of amenities increase, the contribution to the overall value decreases.
However a quick thought experiment is also helpful. Imagine a market in which there are only 3 homes. All are identical, all have identical lots, square footage, bedrooms, quality, and condition. There is only 1 difference between the 3 homes, the number of bathrooms.
House #1 – has no bathroom, at all, anywhere
House #2 – has two full bathrooms
House #3 – has 35 bathrooms
Is the difference per bathroom between House #1 and #2 the same as between house #2 and #3. If you said no, congratulations, you understand the law of diminishing returns and that multiple regression CANNOT be trusted for real estate valuation. If you said yes, please contact me, I have a house to sell you.
In 2017, Town and Country Residential Appraisals reached out to the appraiser community online and asked appraisers to volunteer data from their various areas for us to examine (not to examine their appraisals, only the data that would typically be relied upon for regression). Appraisers from 6 different regions of the country responded. Aside from all 6 data sets failing tests 3 and 4 above, 5 of the 6 data sets additionally showed low levels of confidence in the data that they generated, some offering lower than 10% confidence that the data could be relied upon EVEN IF they had passed all four assumptions above. Please understand, this is not a critique of these appraisers. They delivered to us data that would be used in multi-linear regression. We performed no review of their appraisals or their interpretation of the data delivered (or if they use it at all).
Appraisers can not be complicit in handing over valuation to big data. This has already done damage to the American people and economy, and will only continue to.
There is so much more to say on this subject ie. the importance of P and R squared values, sample sizes, confidence intervals, outliers, etc. However for more reading on this subject, please refer to the following for a primer on these subjects and why linear regression isn’t everything.
In answer to common responses:
- “I only use the data when it gives a logical result.” – This is called confirmation bias. If the confidence interval is low, but the data rendered “makes sense” to you, all you have done is confirmed your own opinion with data that is less accurate than a coin flip in determining contribution (500% less accurate in the case of confidence intervals below 10%.)
- “R Squared values / Sample sizes don’t matter.” – I genuinely want to meet the person teaching people this, as I’ve heard it spouted enough with confidence that someone claiming mathematical competence must be teaching it. Simply, yes they do. I have yet to meet someone who can articulate a mathematical defense of this position, however I think they have the following assumption – “Since we have 100% of the sales data for an area, we have 100% of the sample and therefore, R-squared becomes obsolete.” 1) Unless you are also including all off market sales, not even that statement is correct. 2) ML Regression is not claiming to predict amenity contribution of only sold homes in a market but rather ALL homes, of which, typically, only small percentages sell, meaning that we very much need to consider the R-squared value and its effects on homoscedasticity and normalcy of distribution (tests 1 and 2 above) as well as the sample size and corresponding P value.
By Brianne M Brown, Certified Residential Appraiser and Jesse Ledbetter, Certified Residential Appraiser at Town & Country Residential Appraisals, LLC. Brianne started in the real estate industry in 2000. She met Jesse when she hired him as a trainee back in 2016. Since then, they have gotten married.
The latest BS is Desktop/ADI products put out by the AMCs based on their data mining over the last few years.
Really good article, as a rural appraiser working in markets where values range from 100K to multi millions the CU has been a major detriment
As a General Appraiser with very good access to data, I use regression on that side of my practice and I know how to use it. I more importantly know when not to use it. Further, there is 1 commercially accessible for residential appraisers that uses many other statistical tools not just regression out there and I too use this on my residential side. The point here is really IMHO yes regression can work, but you have to know when and how it’s appropriate – just like any other tool you have in your kit.
So true, I guess that is why we say that “appraising is both an art and a science”.
There are three kinds of lies: lies, damn lies, and statistics. – Benjamin Disraeli
In researching for an assignment I’m working on, and within my market parameters (Detached, +/- 25% GLA, within 1 mile, with HOA fees, etc.,) I pulled up 151 on the surface like properties, the problem is my subject is of a condo form of ownership, thus after filtering I truly only have 14 like sales of which 4 or 5 are good comparables.
My point is, homeowners often don’t know what they own, agents often don’t know what they are selling, lenders often don’t know what their lending on, public officials often don’t know how to correctly categorize a property, and software engineers often don’t know how to engineer.
Seek the truth.
Note the time of the attached article. 2002. Appraisal has been taken over by techno wizards who promise accuracy on a product that only uses math to opine on value; all other activities performed by professional appraisers relies upon their experience, education, ethical conduct and adherence to USPAP which is being abandoned to seduce the remaining appraisers who believe magic is involved in what we do.
It is not magic; it is hard work using the tools of our trade, understood and practiced with objectivity and attention to the behavior of the market participants. This can’t be understood without inspecting the property as in the latest iteration of a valuation, or an algorithm written by a nerdy guy with a new gadget. Get real people, stop buying the crap people are selling appraisers while selling us out to the lowest bidding AMC who is basically a corrupt middleman.
In every market there may be winners and there may be losers. Credibility of agency has no limits in this regard, being the likely root cause of some data discrepancy. The sold examples may present as illogically varied valuation indicators within data sets. They do not necessarily imply the need for alteration of reasonable market derived estimates of worth, simply because the data is not matching up. They skew data for other examples.
Credible market value appraisal for an individual unit requires individual unit comparisons, and a human to make quality judgments about the material worth condition and volume. Credible estimates of worth differences expressed in the form of an applied grid adjustment number usually tie an analysis up nicely. Such analysis approaches do not require subscriptions to anything other than MLS data. Appraisers paying for typing services or whipping out boilerplate language are likely missing the mark for logical individual analysis and competent unique language expression describing why. If appraisers are actually interpreting data it should be just another single step on the field to then type that unique interpretation effort into the report.
The tech industry has developed pretty interesting price analysis programs lately. We’re still waiting for something which accurately expresses value.
Mortgage Brokers are now offering to check a property’s WAIVER status for both listing and selling agents BEFORE the property even goes into contract.
Its a CLOWN SHOW.
Interesting. Have you seen this advertised in standard company materials yet?
https://www.fanniemae.com/content/fact_sheet/property-inspection-waiver-fact-sheet.pdf
With the appraisal waiver program, appraisers will only get difficult and higher risk assignments. So the new deal is nothing under 400k for well qualified borrowers, no more everyday condos, no more full service refinance requests for the better half of well qualified borrowers with decent ltv’s.
https://www.fanniemae.com/singlefamily/property-inspection-waiver
Generally speaking, most appraisers will no longer be asked to provide regular appraisal services in neighborhoods we could actually afford to live in.
Very impressive article, if you like all the pontificating and use of large words and statistical terminology. Yes, I’m Quite sure the author’s PhD friend could make minmeat of 99% of appraisers who use regression analysis, in a courtroom setting. With all the naysaying, I didn’t read any positive recommendations. It was like having an appraisal reviewed by a reviewer with a chip on their shoulder. I gather the author likes the matches pair analysis over MLRegression but doesn’t say so. Oh by the way, matched pairs isn’t absolutely foolproof either. How about a little positive and constructive recommendation instead of the negative ranting about the ills of MLR?? Or is the reader to assume that they have to be a Princeton PhD if they even attempt to use MLR?
I’m looking at somewhat subjective prices of best matched individual sold comparables, then extracting value from that through a myriad of logical line item adjustments on the fly. Using these advanced softwares seems like more trouble than they’re worth. Sounds like I’d be applying routine data clean up methods to an entire data set rather than just an individual spread of comps. The act of selecting good matching comps is how appraisers can bypass a constant need for market derived adjustments in many scenarios. If they’re reasonably aligned, all it takes to adjust pricing to a streamlined value indicator is simple logic through individual line item comparisons, and not all that much extra effort.
It’s important to define the need. Regression may be better for those dealing in mass data, but may not be best for individual comparisons. Manually orientated appraisal development is highly defensible, relatively simple and routine once an appraiser can make short work of data cleanup.
Logical adjustment is easy if you understand the product being appraised for value. A home. A collection of physical components of various price, quality, and volume. Components with real world price tags. Mass data may miss the mark because it can not sort individual details quite like a human can. Then apply logical thoughtful individual adjustments, or not apply them, whatever may lead to credible assignment results.
When a shopper sees a ruined deck, the first question they ask is not what’s the impact of market value of this item. The question they ask is how much is that going to cost me to correct? Smart home shoppers keep a mental checklist of expected costs, then they’re on to the next showing to do that all over again. Some are in it for speculation, others want them turn key. Regression allows too much room for quality of condition error and instead must rely on price indicators for just about everything. These price analysis programs are branded as value results, so just run with that I suppose.
Consider: Using paired sales is regression with two data points. Using aggregated sales data for pairing is linear regression on a single variable while ignoring the “noise” of the other variables (we assume they cancel out).
Do not confuse statistical regression on a large disparate data set (what the article essentially addresses) with regression on appraiser “curated” data sets. Conventional statistical reliability metrics rarely work even if we ignore the argument that the data IS the population and is not a sample set of the population.
The R value on the two paired sales is 100%. The R value of three, four or five matched pairs may also be high, but those three, four or five matched pairs may not be statistically reliable even though matched pairing is the “gold standard” in deriving adjustments.
So yes, some of the parametric statistical software packages are likely resulting in misleading conclusions even with high Rs and low Ps. However, using regression software for appraiser curated data is as reliable as the conventional methods discussed above. We are instead relying on non-parametric statistics, and of course judgement.
I might also add that regular regression analysis is not what FNMA is doing. They are taking our XML data and “chaining” appraisals together. Essentially the subject of one appraisal is the comp in another appraisal and so on up and down the line. By analyzing this chain they are deriving ranges of adjustments. It is the ultimate in appraiser curated data.
Author here: Thanks for the positive comments and additions to the conversation.
Clearly touched a nerve with some, only one thing I’ll comment to that: the point of the article is narrow: Under what circumstances can we trust regression? This is a math question, not an opinion-based discussion – we can trust regression IF it passes the 4 tests above. IF your data set is not tampered with and fulfills the four tests above, then, Yes you can trust the output.
However, IF 1) the data doesn’t pass those four tests, 2) the results fail basic mathematical rigor, then the answer is No. If you chose to use data/results that are not mathematically supported, and claim that you’ve done due diligence, you’ve actually been negligent in your duties. Given that statistical analysis is not a required course for appraisers, and that many new companies are putting out “products” that offer to do the math for you, its important that we educate ourselves on the way these tools work and how/when to use them and when to set them aside.
The question is vital to appraisers: Can/Is regression producing CREDIBLE results. Perhaps in a future series of blogs, we can cover the various tools that our friend the PhD recommended given that the most normalized of data sets in our region could not be used to produce credible regression analysis.