County Assessors’ Standards | AVM Final Rule Guidance
The OCC, FDIC, NCUA, CFPB, and FHFA (collectively, the Agencies) are adopting a final rule to implement AVM quality control standards mandated by the Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act). The quality control standards apply to mortgage originators and secondary market investors in determining the value of a dwelling that is the collateral for mortgage financing.
Under the final rule, institutions involved in specific credit decisions or securitization activities are required to adhere to quality control standards designed to: (1) ensure a high level of confidence in the estimates produced by AVMs; (2) protect against manipulation of data; (3) seek to avoid conflicts of interest; (4) require random sample testing and reviews; and (5) comply with applicable nondiscrimination laws. However, a critical flaw in the rule is its lack of specific parameters or guidelines that define what constitutes “a high level of confidence” in the estimates produced by AVMs. This vagueness raises concerns about how regulatory agencies can effectively enforce a rule without clear standards or criteria. Without defined parameters, the concept of confidence in AVM estimates becomes subjective, making it difficult to ensure compliance, consistency, and enforcement of the standard across the industry.
The Appraisal Foundation (TAF) shares this concern. Their Industry Advisory Council’s task force produced White Papers in 2022 and 2023 to address these issues. These reports echo the need for well defined parameters for testing AVMs to ensure their reliability and effectiveness.
Some highlights from the White Papers:
Consistency in measurement and reporting of AVMs must be developed. Most AVM model vendors self?report a “Confidence Score” with little transparency. A standardized Confidence Score or other measure of uncertainty that can be understood and tested must be developed and adopted by the industry.
AVMs are the complex resultant of the integration of sophisticated methodology, complicated mathematical algorithms, advanced computer science technology and large databases, frequently beyond the understanding of most Americans, including lender underwriting staff. Accordingly, we recommend consideration of a Certification for AVMs meeting a minimal set of standards, which can provide some assurance to non?technical users of AVMs.A list of minimally required reporting elements needs to be developed and adopted for all certified models.
The AVM reporting and testing group determined that the gap in most need of addressing is the lack of an easy to understand and consistent AVM confidence score.
This raises a critical question: how will The Agencies enforce the recent AVM final rule without defined parameters for measuring accuracy and confidence?
The Agencies should look to the example set by County Assessors (Mass Appraisers). AVMs are very similar to the Computer Assisted Mass Appraisal (CAMA) models used by County Assessors. In fact, the International Association of Assessing Officers (IAAO) often does not differentiate between the two. According to the IAAO’s Standard on Mass Appraisal of Real Property, the principles defined here should also be relevant to CAMAs or automated valuation models used for other purposes, such as mortgage portfolio management. And from a July 2014 IAAO feature article: AVMs are also commonly referred to as CAMA (computer-assisted mass appraisal) models.
AVMs and CAMA models are fundamentally similar. Both rely on mathematical models and statistical testing to estimate property values. Consequently, regardless of their intended use, they can be subjected to the same statistical accuracy tests.
County Assessors adhere to rigorous standards for the mathematical models they use, including precise parameters that model results must meet. For example, the effectiveness and accuracy of the CAMA systems are assessed against established standards for ratio studies and mass appraisal set by the IAAO. The transparency and public availability of these standards enhance confidence in the assessment process by enabling results to be independently verified and replicated.
In Washington State, as is common in many regions, all CAMA models used by counties undergo annual testing. This testing is documented in the Department of Revenue’s yearly report titled “Measuring Real Property Appraisal Performance.” The report provides detailed statistical measures, including Sales Ratio, Coefficient of Concentration, Price Related Differential, Median Percentage Deviation, and Coefficient of Dispersion, among other measures to assess and ensure the accuracy and effectiveness of these appraisal models.
Some examples from the report:
According to the IAAO Standard on Ratio Studies, the median is the appropriate measure of central tendency for monitoring appraisal performance. The IAAO standard states that the median ratio for all assessments should be between 0.90 and 1.10.
Another aspect of uniformity is the treatment of properties of different values. The price-related differential (PRD) is a statistic used to measure whether high-value properties and low-value properties are assessed at the same ratio to market value.
IAAO Standards represent a consensus in the assessing profession. They released the first Standard on AVMs in 2003 and updated it most recently in 2014. In developing the standard, mass appraisers experienced in CAMA model-building focused on model calibration and specification, including the application of the three traditional approaches to value in an AVM environment. Experts in ratio studies and quality assurance extracted relevant information from the IAAO’s Standard on Ratio Studies and Mass Appraisal of Real Property for the AVM standard. The IAAO also reached out to private-sector parties, such as the Mortgage Bankers Association, and private sector individuals with industry expertise, who could truth test the standard. One stated goal in developing the standard was that the product be applicable in a wide variety of property valuation disciplines and purposes beyond the assessment profession. In addition to involving private-sector industry representatives to participate and provide feedback, the committee used terms more familiar to appraisers than to assessors.
The definition from the IAAO’s Standards on AVMs highlights two critical components that underpin the reliability of AVMs and/or CAMA models:
Data Integrity:
The principle “garbage in, garbage out” is particularly relevant to AVMs. If the input data is flawed—whether incomplete, outdated, or inaccurate—the resulting valuation will likely be unreliable. High-quality data is crucial for accurately reflecting the intricacies of the real estate market, including local trends, property modifications, and seasonal pricing fluctuations. Human appraisal expertise plays a vital role in selecting the appropriate data, as experienced professionals can identify which data is most relevant and indicative of market conditions.
Modeler Expertise:
Although AVMs rely on automation, the insight and experience of human modelers are indispensable. The modeler is responsible for choosing suitable modeling techniques—such as linear regression, decision trees, neural networks, etc.—and ensuring the model accounts for diverse market conditions and outliers. Skilled professionals not only interpret the results but also assess the model’s strengths and weaknesses, making continuous improvements to enhance its precision. Furthermore, the knowledge of seasoned appraisers is crucial in validating the results, ensuring the AVM produces credible results.
As we consider this issue, it’s important to recognize that County Assessors and their CAMA models typically have distinct advantages over AVM developers. Assessors typically have access to comprehensive and granular data on every sale within their specific market areas. They can also deploy staff for on-site property inspections, which adds an additional layer of accuracy to their assessments. The inherent differences in data access and inspection capabilities between County Assessors and AVM developers contribute to the general observation that AVMs typically produce less accurate valuations than County Assessor CAMA models. County Assessors have the advantage of being able to calibrate their models specifically for one market, allowing for a more tailored and precise approach. In contrast, AVM developers must design their models to function across a much broader, often nationwide area which increases the risk of model misspecification. These disparities could lead to AVMs being less accurate and more prone to producing regressive valuations compared to the CAMA models used by County Assessors.
However, a direct comparison is challenging because AVMs are typically “black boxes” with proprietary algorithms and data. Due to the absence of standardized benchmarks, they cannot be directly compared to County Assessor models or even each other. Consequently, any evaluation of their relative performance remains speculative.
These concerns are further amplified by a recent study conducted by the Federal Reserve Bank of Philadelphia, which highlights the tendency for mathematical models to produce regressive results. Historically, studies have shown that property assessments can exhibit a regressive pattern, where less expensive homes are overassessed compared to more expensive ones. The Federal Reserve report attributes this phenomenon to two primary factors: 60% is due to flawed valuation methods, while the remaining 40% is attributed to infrequent reappraisal.
According to the author of the Federal Reserve report:
I find that infrequent reappraisal can explain approximately 40% of the true degree of assessment regressivity… infrequent reappraisal explanation, which takes two forms: (1) revaluation cycles that span more than 1 year and (2) assessment growth limit laws. The remaining 60% can be explained by flawed valuation methods. I show that there is a clear relationship between assessment regressivity and valuation model misspecification.
In conclusion, there is an urgent need for well-defined standards and parameters for AVMs. Without clear guidelines, the AVM evaluation process is subjective and open to interpretation, making it difficult to ensure consistent and reliable valuations. Establishing defined standards for AVMs would help align their accuracy and reliability with that of County Assessor models, ultimately bolstering stakeholder confidence in these automated tools. The absence of such standards not only undermines the purpose of having an enforceable rule but also risks perpetuating inaccuracies and biases in property valuations.
This makes too much sense. It’s another rule that the regulators essentially told the banks and lenders to enforce. Just like they told banks and lenders to enforce AIR, Customary & Reasonable Fees, and to audit their AMCs to adhere to these items. That didn’t happen, and neither will this.
Communist and Retroactive Fees …
For example, the effectiveness and accuracy of the CAMA systems are assessed against established standards for ratio studies and mass appraisal set by the IAAO. The transparency and public availability of these standards enhance confidence in the assessment process by enabling results to be independently verified and replicated. NOT – This is complete BS – All CAMA models use the “Cost Approach” with little or no market support. Techniques that may include but are not limited to, goodness-of-fit statistics, and model performance statistics such as appraisal-to-sale ratio studies, evaluation of hold-out samples, or analysis of residuals. Find me an unlicensed “Assessor’ administrator who even begins to understand this. IAAO guidelines on AVM’s fill one page of their (AAO Mass Appraisal Standards approved July 2017).
What gets me thinking is the fundamental modeling shifts. Assessment systems are subject to oversight by the citizens. Imagine our entire countries assessment systems being administered by one set of private corporations. Nobody voted for a centralized model of value control which has the potential to reshape price and value benchmarks throughout the entire country. A usurpation of the entire system of democratic feedback and free market response. Maybe Mr Kiedrowski could please chime in on that point. Proportional to the appraiser or realty community only a small fraction of licensed persons have been trained on the underlying mechanisms which govern mass appraisal and/or avm type systems. As you said, up to this point there has not been much attention paid to the matter of avm guidelines either. Great points Dave, well informed.
And there it is again. I swear, I saw it. “Determine value”. Right there in the first or second paragraph. Hard to believe. News flash, we don’t determine value. The only two things that determine value are the courts and the market, regardless of what the rest of sentient existence claims. It’s a little tidbit to keep in mind.
Lenders and avm providers will be self certifying inflated values from heavily tainted data pools and will adjust proprietary avm programming on the fly with no external oversight. There is less than one year before this program runs full speed. Lender program participation will be mandatory. Read the final rule with commentary, first link, 190 pages. The conflict of interest gap is incredible. ‘adjustable cascading rule sets’.
https://www.consumerfinance.gov/about-us/blog/cfpb-approves-rule-to-ensure-accuracy-and-accountability-in-the-use-of-ai-and-algorithms-in-home-appraisals/
As modeling technology continues to evolve, this flexible approach will allow institutions to refine their implementation of the rule as appropriate. The proposed and now adopted approach will allow mortgage originators and secondary market issuers the flexibility to set their quality control standards for covered AVMs as appropriate based on the size, complexity, and risk profile of their institution and the transactions for which they would use AVMs covered by the rule.
the agencies have determined that a flexible approach to implementing the quality control standards would allow the implementation of the standards to evolve along with AVM technology and reduce compliance costs. Different policies, practices, 81 procedures, and control systems may be appropriate for institutions of different sizes with different business models and risk profiles, and a more prescriptive rule could unduly restrict institutions’ efforts to set their risk management practices accordingly. For these reasons and after considering the comments, the agencies are not issuing additional guidance at this time and recommend that institutions review and consider existing guidance when establishing and implementing appropriate polices, practices, procedures, and control systems for AVM quality control
As noted, the final rule implements a statutory mandate, thereby limiting the ability of covered agencies to consider alternatives. That said, agencies did exercise authority provided by section 1125 to include the nondiscrimination quality-control factor (given continued evidence of disparities in residential property lending terms along racial and ethnic lines) the size of their institution and the risk and complexity of transactions for which they will use covered AVMs—institutions should be able to work with AVM providers to assist them with their compliance obligations under the rule
Welcome to the used car lot, formerly known as real property appraisal!
Another false flag Baggins! Most assessments country wide are tanking at 50% – the assessment community will guess using a fantasy cost approach and it will take them 2-3 years! The lending community is just PO that they can no longer use an assessment in place of a proper appraisal. They are also using Zillow now, but they almost went bankrupt with their AVM requiring a complete reevaluation of traditional software development. Let’s not forget Lyle Rieke!!!!
Not sure what you’re referring to with a false flag. The content above, everything after the link is copied language from the CFPB’s own avm documentation. / In my state, assessors state wide did adjust up to match higher market values. Peoples taxes and insurance costs increased substantially.
https://commonsenseinstituteco.org/colorado-property-tax-primer/
County assessment data has never been a reliable basis to substitute live human appraisers. Formulas for assessment, rate, and levy, are all subject to change based on political and other external forces, citizens get to vote on those issues. Now, avm derived ‘market values’ will be subject to cryptic difficult to understand or recreate algorithmic calculation adjustments, and nobody will get to vote on the matter.
This is a significant departure from the rules which govern human appraisers such as re creatable methods, recognized methods, requiring documentation of the method, furnishing a workfile. Where is the individual accountability?
‘Interested parties’ will be in control of the avm systems. Why bother keeping up the pretense of separation from loan production, when the loan production team will now be in charge of governing rule sets which directly influence avm systems end value data outputs?
No more humans complaining about value pressure. ROV’s will be a thing of the past. Maintaining maximized production volume will be the measurement of the avm’s operable state. The automation and behind the scenes data manipulation will result in a continuous pump and dump predatory investment cycle. Central planning never works.
This blogs article brings forward great suggestions for minimum avm standards and uniform rules to govern the system. Unfortunately if that’s what lenders wanted, they’d simply continue using human appraisers whom are already subjected to more than adequate regulation and oversight.
What exactly is wrong with lower housing prices? The premise of moving away from human appraisers and towards these automated systems is that people deserve higher prices. This is the conclusion of the PAVE presidential task force; people should pay substantially higher housing prices, aka; ‘housing value equity’. Unlike zillow, taxpayers will be backing this one. All gse loans are backed by FDIC insurance.
Can you say all data owned by CORELOGIC with NO reliable analytics and of course that would include their dinosaur Marshall & Swift! CLA adjustments are not a substitute to FMV 2024 based on sales; and not a reliable update because it is simply a multiplier time the original bogus CAMA system. Model performance statistics such as evaluation of hold-out samples, correlation analysis or analysis of residuals are just words to the assessing community – way above their pay grade!
Newsflash, the data input is always incorrect because the properties are not inspected and or measured. In forty years I’ve seen less than a handful of houses wherein the sf was within five sf of actual. It’s all political and assessors are going to assess houses for what they’re told to assess houses for, regardless of laws, rules and regulations. And, what do bankers have to do with tax assessing except to utilize as another step to get rid of actual appraisers? The appraisal districts in Texas have been a thorn in the side of actual appraisers since 1981 and the public’s and lender’s understanding of same is a bad in 2024 as it was in 1981.
When most of the information they’re using is wrong (mls and/or assessors records) how can an AVM be relied upon EVER? Stats and AVMs for real estate are a joke when the information they gather is so extremely flawed.
So – Deny analysis of data because it may not be completely accurate. The buy and sell sale price is fact. The use of the “cost approach” requires NO verified factual data. Go figure fellow LICENSED professionals!
I brought this up because it mentions how important accurate data is. Wasn’t necessarily thinking of the CA when I typed that out.
However, the buy and sell prices are NOT always fact. In my area they are often inflated by agents and I have to call around to figure out what the correct selling price was or buy the deed to have it for my work-file. The last “fact” was inflated by $30,000 and had already been used as the high end sale for that neighborhood by multiple appraisers. It was reported and still not changed.
Throwing all that information in an AVM and having it kick out anything remotely defendable is laughable. Maybe where you are you can trust all mls and tax records. Maybe where I am no one gives two hoots about accuracy in information.
The verification process is one of the most important parts of my job in order to produce a credible report.
So, yes, to me, because of the low to no standards for the information entered, AVMs are a joke.
Maybe I should stop verifying since, when mentioned that most of the information I gather is wrong, people are jerks. No one else seems to care anyway. Maybe I can just look at Zillow?
NAR settlement came and went for many appraisers. But for others it’s a real game changer. The delineation of which side you fall on has to do specifically with your states rules on concessions disclosures. Apparently CO updated some contract forms for a simple one, two, three option check box, clearly describing who’s paying fees on either side. But in other states it sounds like the buyers agency fee may be stated as a concession, and may also be combined with actual concessions, in the concession reporting lines. So this can get real complicated real fast.
Appraisers must identify if they need to be researching this data and following the advice as stated in this latest WorkingRE article on the NAR settlement. Read this in detail, check your own states laws, and identify if you need to be engaging in manual concessions verification for each and every comparable you select. If you need to do that, the solution is obviously to ask your MLS to approach concessions reporting differently, and have more straight forward state forms so that these agency fees do not need to appear in concessions disclosure lines. In some places they are, so be very aware of what’s happening and the importance of this issue. In some places MLS systems removed all agency fee disclosure from searchable data completely.
The implications how this data could be fed into avm systems and create extremely over valued positions is incredible. Like a redundant data loop which leads to eventual stack overflow. One of the most glaring instances of federal rule making incompetency ever.
https://www.workingre.com/nar-settlement-what-it-means-for-appraisers/
CO first state in the nation to put forth AI systems laws.
Specifically mentioned; AI systems which pertain to lending.
Might just be easier to continue using human appraisers instead.
https://www.skadden.com/insights/publications/2024/06/colorados-landmark-ai-act
https://www.cpr.org/2024/06/17/colorado-artificial-intelligence-law-implementation-ramifications/
https://leg.colorado.gov/bills/sb24-205