ANALYSIS OF NORTH AMERICAN SECURITIES ADMINISTRATORS ASSOCIATION’S (NASAA) REG BI SURVEYS

The North American Securities Administrators Association’s (NASAA) Regulation Best Interest (Reg BI) Implementation Committee (Committee) conducted two surveys with the stated purpose of measuring the impact of the U.S. Securities and Exchange Commission’s (SEC) Reg BI and level of compliance with the regulation. While any effort to measure the effect of public policy is laudable, a careful review of the Committee’s research effort uncovered significant deficiencies, including the notable failure to follow eight well-established and critical survey research practices. The flaws in the design and execution of this research mean that its findings and conclusions cannot be relied upon as accurate measures of firm behavior, compliance with Reg BI, the impact of the regulation, or the extent to which the interests of investors are well-served.

Survey research is a mature field, with several well-accepted standards and codes of conduct that have been recognized for decades. Among these standards and codes of conduct are:

1. Questions must be aligned with each survey’s research objective.

2. Questions must be clear and understandable to ensure that every respondent knows what is being asked and each respondent’s answer is based on the same understanding.

3. Answer categories must allow each respondent to accurately report his or her answer.

4. A questionnaire and analysis must be objective and without a pre-determined agenda or effort to obtain a specified result.

5. An effort to measure the impact of an event or change, such as the impact of Reg BI, must: 1) include clear before and after measures, 2) demonstrate causality, and 3) eliminate plausible rival explanations.

6. Sampling of a population must be designed to achieve an accurate measurement of the characteristics of the entire population to which it is applied.

7. Any report of a survey must include detailed information about sampling and associated methodology to allow readers to evaluate this crucial component of the survey process.

8. The results of a survey must be reported accurately and not mischaracterized.

The attached report by Greenwald Research, an established expert in research and survey methodology, points out numerous instances in which these standards were not met. The analysis also found that the surveys failed to collect information on key actions firms have taken that are pertinent to Reg BI and the best interests of their clients. As a result, the conclusions reached by the Committee are not supported by the information gathered in the surveys and should not be considered as a basis for action by the SEC or other policymakers. Any attempt to realistically assess the impact of Reg BI and its impact on investors and investment products to achieve NASAA’s stated research purpose would require a completely new survey or other type of analysis