Collateral Underwriter, What’s Under the Hood?
Collateral Underwriter – Garbage In, Garbage Out (GIGO)
FNMA has a fascination for first patenting garbage, & then implementing it as policy! FNMA simply LIED! Collateral Underwriter IS an automated ‘appraisal review’ despite disclaimers. It was never intended as anything else. Read the patent application below.
Before I get into the details, I want to ask each reader to forward this to every appraiser you know. This concerns FNMA’s patent of the seriously flawed techniques and underlying support for their Collateral Underwriter (CU) Risk Rating “System”.
It’s time for appraisers to take action against the unreliable system which is unfairly hurting appraisers, and misleading consumers and taxpayers into a false sense of being protected from the next crash.
Reference: United States Patent Application Publication Obrecht et al.
Publication No.: US 2015/0302419 A1
Publication Date: Oct. 22, 2015
Inventors: Paul Obrecht of Santa Fe, NM; Adam Davis of Alexandria , VA; Alex Medinets of Washington, DC; Eric Rosenblatt of Derwood, MD; and Zachary Dawson of Washington DC.
Assignee: FANNIE MAE, Washington, DC, Application No. 14/258,512
Filed: April 22, 2014
“ABSTRACT –A system and method for appraisal adjustments scoring rates the quality of adjustments made by appraisers in their appraisals of real property. This is done by accessing a model adjustment database that is based on an automated valuation model and a peer adjustment database that is based on an aggregate of appraiser peers in the same geographic location as the subject property. A comparison is conducted for adjustments made by the appraiser to both model adjustments and peer adjustments to determine discrepancies. When a discrepancy is larger than a threshold, a message or warning may be generated. A sales pool composition database is also accessed to determine a valuation impact of the flagged adjustments given the particular set of comparable sales selected by the appraiser. After data evaluation, a rating score is calculated for the property appraisal based on the number and severity of messages and the valuation impact.”
Okay fellow appraisers, please go ahead and reread the above. It is a verbatim quote. The use of forty word sentences is even hard for ME to follow and y’all know how long winded I get!
I’ll wait right here patiently.
Am I the only one that sees this as an outright violation of Federal Law? Specifically FIRREA 1989 TITLE XI, and Dodd Frank?
Isn’t “rating” the quality of our adjustments in an appraisal, an appraisal review? “Determine a valuation impact?” You mean is the appraisal conclusion credible or not. THAT is a review.
Before I go any further. NOT ONE of the “inventors” currently holds a valid appraisal license from any state or jurisdiction according to the Appraisal Subcommittees Federal Registry. NOT ONE!
There is a possible name match for Mr. Davis in a different City (Arlington) which is adjacent to Alexandria for an INACTIVE license. Mr. Dawson HAD a regular (non-certified) license from Minnesota from 2003 to August, 2010. It is not currently active.
So what we have is at best an illegal appraisal review system, invented by people that are not licensed appraisers. I say illegal because it does NOT conform to the requirements of FIRREA, Dodd Frank or USPAP. And when used by FNMA it IS being applied in Federally Regulated Transactions.
Okay, second sentence. The rating is done by looking at some kind of undefined “model adjustment” database that was in turn developed from AVMs! Wow. THAT must be reliable! Where do we find the statistical variance reliability statement (and amount) for each appraisal reviewed?
The following quote was provided by a friend:
“Page 291, Real Estate Appraisal A Critical Analysis of Theory and Practice by Paul F Went, 1956, published by Henry Hold and Company, New York.
“Various mathematical techniques have been developed for adjusting comparable market sales prices.”
“Although adjustments for differences in the timing of sales and for non-comparability are certainly necessary, it might be contended that the over refinement of mathematical adjustment techniques has frequently encouraged appraisers to lose sight of the basic principles of fixing values by reference to comparable sales. Some of the simpler and more straight-forward techniques for adjusting comparable sales recommend themselves in terms of their logic and simplicity in application.”
“Organized price data are becoming increasingly available for various sectors of the real estate market. This improvement in the quantity and quality of market data should enhance the reliability of the market comparison approach and encourage its more general application. As the knowledge of the buyers and sellers increases and as the market structure becomes more efficient, less attention need be devoted to the adjustment of market sales data. Viewing these developments as worthy objectives, the efforts of appraisal groups can be better given to the improvement of basic market data than to the continued refinement of mathematical procedures based in the last analysis upon judgment and opinion.”
After the AVM comparison, the subject appraisal is also compared with the so called “peer adjustment database” aggregated from appraisers that have performed appraisals in the same geographic area.
IF the majority of THOSE adjustments could be documented to be “market based” in accordance with the FNMA definition of market value at least one half of the system would have potential relevance. Imagine. A system that from the very outset can never be more than fifty percent accurate AT BEST!
The problem is that FNMA implemented this new system BEFORE they advised appraisers that the old ‘guidelines method’ of adjusting individual line items, net, and gross adjustments was determined to have caused many, if not most appraisers to appraise to the guidelines rather than to the market.
It’s significant that FNMA told appraisers in 2015 that they are NOT to make adjustments to old, outdated ‘guidelines’ (such as those adjustments their database is filled with).
We cannot stress enough that their “peer” database of adjustments is made up of adjustments that were made ‘to artificial FNMA guidelines’ through 2014 rather than to market perceptions.
Our work is rated against one metric that is completely unreliable (AVM), and another that is known to be seriously flawed at the outset. It is flawed to the point of being completely unreliable. It’s arguable that it should not be called or referred to as a “peer” database at all. I’m a professional. So are MY peers. MY peers have NEVER adjusted to or appraised to artificial guidelines. MY PEERS have always adjusted to market. And when market derived adjustments fell outside FNMAs whimsical guidelines, we EXPLAINED them! My peers are competent licensed or certified professionals.
Keep in mind that AVMs are ultimately regression based. ALL regression analyses have a margin of error. Or better yet their own “reliability rating”. It’s doubtful that most adequately account for even 70% of value related elements. That means the AVM “model” ONLY “accurately” accounts for $210,000 value factors of every $300,000 sale! Impact? At least $90,000, 30%, is being arbitrarily allocated to OTHER elements for each adjustment. Or arbitrarily excluded from the calculations. If there are only three adjustments to a comparative sale, each one could be inflated by $30,000, assuming all are allocated similar overage. OR one or more factors MAY have been perceived in the real market to be up to $90,000 more – though pre-programmed software has determined it to be otherwise.
Pointing out that virtually every FNMA appraisal written also had a statement of limiting conditions prohibiting its use for any other purpose would probably just be ‘piling on’ of charges.
FIRREA Title XI:
“The Appraisal Subcommittee (ASC) of the Federal Financial Institutions Examination Council (FFIEC) was created on August 9, 1989, pursuant to Title XI of the Financial Institutions Reform, Recovery, and Enforcement Act of 1989 (Title XI). In response to the “Savings and Loan Crisis,” Congress adopted Title XI to address the problem of unregulated persons performing incompetent and/or fraudulent appraisals for federally regulated financial institutions. Title XI’s purpose is to “provide that Federal financial and public policy interests in real estate transactions will be protected by requiring that real estate appraisals utilized in connection with federally related transactions are performed in writing, in accordance with uniform standards, by individuals whose competency has been demonstrated and whose professional conduct will be subject to effective supervision.”
I don’t see where FIRREA excludes automated appraisal reviews performed by faulty algorithms using AVMs and inapplicable databases. An appraisal review IS an appraisal.
A system that ‘rates’ the adjustment portion of an appraisal is an appraisal review system regardless of how it is named. Just as an AVM prepared by an appraiser is an “appraisal.” A system that is inherently flawed that performs automated review appraisals is one that is not ‘competent’ to the task. It is “incompetent”. Further to knowingly use such a system; and to make false claims about its reliability or applicability is fraud or at least gross negligence.
Appraisers have been told repeatedly that the Collateral Underwriter is NOT an appraisal review, and that it is NOT an automated appraisal review based on AVMs! Reading the patent abstract, THAT claim is completely false.
USPAP 2014-2015 (In effect when Collateral Underwriter was implemented)
“APPRAISAL REVIEW: the act or process of developing and communicating an opinion about the quality of another appraiser’s work that was performed as part of an appraisal or appraisal review assignment. Comment: The subject of an appraisal review assignment may be all or part of a report, workfile, or a combination of these.”
There are a series of undisclosed assumptive opinions inherent within Collateral Underwriter. The inventors’ opinions as to what is deemed applicable comparison and what an acceptable threshold is, and assumptions of applicability for their algorithm. It’s interesting that no peer review of the Collateral Underwriter process is documented in the patent application. WE were left to assume the unlicensed inventors are competent.
The input of previously low level licensees (not certified) with less than seven (7) full years’ experience who in turn are not competent to perform complex appraisals themselves, develop extremely complex appraisal review processes to review the work of those that ARE competent? No wonder a description of Collateral Underwriter had to be so carefully parsed that it could spuriously be argued to NOT be a review!
Whether it is a human being, or a programmed “process” that was developed by human beings to “rate” portions of an appraisal, rating or opining as to the quality of an appraisers work is in fact an appraisal review.
In this instance, Collateral Underwriter is a review BY incompetent (unlicensed) people. People not qualified to perform the specific appraisal reviews being conducted in bulk, using methods that are not recognized sound appraisal practices and reporting the results of that / those reviews via pre-determined generalized messages. Further, it is a ‘review’ using databases AND methods that are known to be unreliable throughout the entire profession.
Lastly, it is a review that provides a misleading result. Individual adjustments developed by the software imply a degree of accuracy that is not feasible. Therefore if the stated accuracy is not possible, why should ANY part of the conclusions be considered valid? It’s more than mere failure to round results to “market significant” numbers. It is the misuse of completely unsupported cumulative automated adjustments that are based on mathematical formulas rather than actual market perceptions of human beings that buy and sell real estate.
Any LICENSED or CERTIFIED appraiser that performed to the above ‘standards’ FNMAs system uses would lose their license.
Interim Final Rule Highlights
Federal Reserve System
Truth in Lending
AGENCY: Board of Governors of the Federal Reserve System
TILA (REG. Z) established new requirements for appraisal independence for consumer credit transactions secured by the consumer’s principal dwelling. To ensure that real estate appraisals are based on the appraiser’s independent professional judgment, free of any influence or pressure that may be exerted by parties that have an interest in the transaction. (Page number 1)
Compliance date is April 1, 2011.
Part of the appraisal independence provisions are: Prohibit coercion, bribery and other similar actions designed to cause an appraiser to base the appraised value of the property on factors other than the appraiser’s independent judgment. (3)
These provisions are contained in TILA Section 129E, which applies to any consumer credit transaction that is secured by the consumer’s principal dwelling.
FNMA is a purchaser of loans secured by the appraisals they are ‘rating.’ That’s definitely an interest in the transaction. Rating adjustments and then communicating “their acceptability” based on a rating system is both an appraisal “review.” It is also both influence and pressure on an appraiser(s) when those ratings or messages are communicated to them. Whether allowed under their (FNMA) licensing agreements with lenders or not, the fact remains that SSR / Collateral Underwriter ratings and messages ARE being communicated to appraisers. These in turn cause them to believe they must either change their adjustments “to conform” to spurious “peers”, and or that they must either now use or explain why the Collateral Underwriter identified potential comparable sales have not been used. All of these things cause an appraiser to consider (or base) the appraised value on factors other than their own independent judgement.
Please see the PDF Patent Application as I received it below.
- Note the cover sheet and page 1 of 10 exemplary sales ‘comparison’ grids both omit unadjusted comparable sale prices. Why? Would inclusion provide visual evidence of how far off the process is?
- Note all ratings are pre UAD: Good, Excellent, Fair, or Y (yes), N (no). Yet this same Collateral Underwriter is used to ‘review’ or rate UAD format grading. How can an appraiser be ‘outside’ peer adjustments when FNMA’s database uses pre UAD subjective comparative property ratings that do not universally match or equate directly to current UAD absolute ratings?
- Sheet 1 of 10 inclusive show the order of the steps within the process. Good or bad, they are worth reviewing.
- Page 1 of 9 makes several statements about Comparative Sales Analysis. Several of these are either only partially true or unsupported within the document. Paragraph  correctly identifies sales comparison elements but neglects the impact of the other two traditional approaches to value completely. It ignores any benefit they may have as a checks and balance against potential erroneous results from the sales approach. The paragraph summary sentence states “The resulting opinion should represent the appraiser’s professional conclusion, based on market data, logical analysis, and judgement.” NOTHING in the patent application authors’ own descriptive sentence suggests substitution or omission of ANY of these three criteria could produce a reliable result.
- Paragraph  states the appraiser documents the facts concerning the subject and comparable sales. It makes no mention of how the ‘facts’ used in the AVM OR peer databases are verified. Or whether they ARE verified beyond the ‘fact’ that they have been accepted by FNMA.
- Paragraph  has the appraiser calculating adjustment amounts with no mention that those “calculations” are derived from data taken from the marketplace OR identifying how that data is taken.
- Paragraph  is where the offensive underlying suppositions start.
“Many appraisers however tend to under adjust in their appraisals.” “Routinely select superior comparables and then fail to subtract an appropriate amount?” My favorite? “Furthermore, because appraisers routinely attempt to create the best impression of a subject property, the set of comps that they select and the adjustments that they make can create a false and inflated value.”
Perhaps the last statement best explains why none of these ‘inventors’ have a valid appraisal license anymore. Is this stated as one of the underlying extraordinary assumptions in all Collateral Underwriter ‘reviews’?
- Paragraph  is as offensive as number 7. Appraisers inflate property values.
- Paragraph  is the authors’ statement that
“…there is a need for a system and method to rate the quality of the adjustments made by appraisers in their appraisals of real property. Furthermore, there is a need for a system and method for detecting and quantifying adjustment issues found in an appraisal.”
If ONLY someone had told the inventors about FIELD REVIEW APPRAISALS! What’s really surprising is that old timers at FNMA used to know what field review appraisals were for.
- Paragraph  admits the system is not merely a generalized rating to identify heightened risk, but is rather a system and method of “evaluating risk in the adjustment of the comps in real estate appraisal.”
“Using algorithmic modeling various embodiments evaluate appraisers claims against industry standards, model predictions, and geographic information service (GIS) analysis, etc. Various embodiments detect erroneous adjustments because they are both materially different from a model estimate and materially different than those made by the majority of appraisers in the same area.”
I am outraged by both the above assumptions, and logic behind the conclusion that being different than a majority of appraisers in a given area equates to being in error! Twenty low paid AMC FNMA appraisers do cursory work with rote “safe” adjustments and then a COMPETENT and HONEST appraiser comes along and finally does a decent job of extracting real market data and THAT is the one that is identified as having “erroneous data” or adjustments! When FNMA is notified of this and the ‘suspected evil appraiser’ proves his adjustments, does FNMA then go back and correct the database or do they leave the erroneous data there for repeated use? Do they then pull the 20 loans that were purchased based on the bad rote appraisals and make the originators buy them back?
- Paragraphs  and  are equally offensive. Paragraph  introduces the comparable sales pool. Paragraph  identifies the invented application as being able to be “embodied” in a variety of diverse unidentified business processes, computer implemented methods, computer program products, computer systems and networks, user interfaces, application programming interfaces, and the like.
- Pages 2 through 7 of 10 give more details concerning the first set of illustrations about the process.
The Uniform Standards of Professional Appraisal Practice, or USPAP, provides for mass appraisal standards to govern the use of hedonic regressions and other automated valuation models when used for real estate appraisal. Appraisal methodology treats the hedonic regression as essentially a statistically robust form of the sales comparison approach. Hedonic models are commonly used in tax assessment, litigation, academic studies, and other mass appraisal projects.
The FNMA Collateral Underwriter system purports to accurately analyze value differences in site area. In order to do so credibly, it would also have to ‘know’, determine, or consider the total site value. The same holds true for depreciated value of living area, as SF or rooms. IF these are known, why doesn’t the ‘process’ also calculate and report the replacement cost? Certainly ‘Big Data’ has access to Marshal and Swift, Means Residential Cost Estimator, and or Building-cost.net.
A review that does not even cross check the validity of its results with another traditionally accepted appraisal technique is not well supported. Is the concern that the vaunted process results would be contradictory if subjected to multiple appraisal techniques?
It’s true that in residential appraising there is often not enough available data to adequately develop an income approach using gross multipliers. FNMA on the other hand processes hundreds of thousands of transactions where rent surveys and gross multipliers have been determined for property all across the country. If the patented Collateral Underwriter process is so reliable, why can it not also be cross checked using the Income Approach as applied to small residential properties?
FNMA’s policies and processes exemplify circular thinking that undermines the validity of real estate appraisal itself. And diminishes the trust that Americans have or will have in the appraisal of real estate.
FNMA decided it would not always require a Cost Approach to be completed, accepting the premise that depreciation from all causes in older improvements is ‘difficult’ to determine by individual appraisers. THIS policy was not updated with the development of their Collateral Underwriter process huge databases. Neither was inclusion of the income approach.
In short, FNMA has developed a system or process of comparative line item valuations based on:
- flawed ‘peer’ databases and
- a process that ONLY considers ONE of the THREE traditional approaches to value!
How does THAT process meet the requirements of FIRREA?
I urge those with particular expertise in developing regression models to analyze this process and offer their opinions.
I’m not such an expert, but I HAVE read the fable about The Emperor’s New Clothes!