Survey Members the Right Way

Survey design is a skill and, in a perfect world, everyone would hire research professionals. But time and budget constraints lead association staff to conduct surveys themselves. Here are a few basic ways to make sure that your DIY surveys yield solid information you can act on.

Most REALTOR® associations conduct surveys. Member feedback is critical for strategic planning and program and service development. In fact, surveys can measure and quantify just about anything, but should always be undertaken with a specific business need or question in mind. Approach survey building carefully so the data you get back can guide the decisions you need to make.

For example, let’s say your association wants to re­examine “the way things have always been done” and create new value for your members. A survey can help you figure out how to create the best possible programs and services for your members. So where do you start?

There are several steps to take before writing survey questions:

1. Know what question you need the data to answer. Let’s say you want to find the one new service that would most benefit members right now.
2. Know what you’re going to do with the data. The information you get back needs to be actionable. In this scenario, the data will guide your program development, so be sure that leadership is on board with following where the data guides you.
3. Define the problem. What issue will the survey results help you resolve? In this case, there’s a perceived lack of value in member services.
4. Put together a research objective. A research objective should be very clear and simple, and phrased: “To determine or measure X in order to Y.” Here, the objective could be, “To determine what new service would be of most value to our members right now in order to create a program that will increase the value proposition of membership.”

Once you’ve taken these initial steps, you’re ready to write your survey questions. The major things to keep in mind when designing a survey are that you want it to:

1. Be easy for your members to answer honestly and with as little of their time and energy as possible.
2. Reflect as little of your own bias as possible (and we all have biases).
3. Reflect the group you’re trying to draw conclusions from (in this case, all of your members).
4. Enable you to draw conclusions from the data in terms of who your members are and what they need.
5. Yield actionable information for your association.

In our scenario, your questions should address not only what members want and need but also which association services members are aware of.

Surveys Dos

1. Give your members an “out.” If there is any chance that someone might not have an answer or opinion about a question, include “Don’t know” or “No opinion” as responses. Allow your members to respond honestly. Not allowing for these neutral responses will skew your data and make your results unreliable.
2. Ask one question at a time. For instance, if you want to know which services your members are actually aware of as well as how useful they find them, ask these questions separately. Ask only those who are aware of a particular service to tell you how useful they find it. If you ask your members how useful they find something they didn’t know existed, you’ve skewed your data.
3. Make your response lists exhaustive. For every question, think of as many responses as possible and use those as your options. Include a final response option of “Other, please specify” with a single-line comment box to cover anything you might not have thought of.
4. Use skip patterns. Even the most basic survey software will allow you to program a survey so that people who answer one way to a particular question will then get another question that applies to them, and people who answer another way won’t see that question. Skip patterns ensure that you don’t waste your members’ time with questions they can’t answer, and that they aren’t giving responses that make no sense. If you’re using a hard-copy survey, you can write, “If no, skip to question X.”
5. Add at least one open-ended ­question–but use no more than three. Your members will always want a question at the end of the survey that lets them tell you what they think in their own words. “If you have any further comments, please let us know here” with a comment box underneath is a great way to end a survey. Don’t offer an e-mail address for them to send comments; build commenting into your survey. Keep in mind that it’s very difficult (sometimes impossible) to quantify the responses to open-ended questions. If you find yourself wanting to ask many open-ended questions, you shouldn’t be doing a survey because you don’t yet know what you need to measure. Back up and talk to your members via ­qualitative research, such as telephone interviews or focus groups.
6. Phrase the questions in a neutral way. Take a hard look at your own need to hear particular answers, and make sure that doesn’t come through in your survey. Resist the urge to explain why a particular service is important, for example, if what you’re trying to measure is how important your members think it is.
7. Avoid yes-or-no questions. There are very few cases in which your members will have responses as black and white as “yes” or “no” to anything. You will get a much more realistic idea of how your members feel about something using a rating scale. For instance, instead of asking, “Do you find X member resource useful,” consider asking them to rate how useful that resource is on a scale of 1 to 5, where 1 is “not at all useful” and 5 is “very useful.” If you must ask a yes-or-no question (for instance, whether your members support a particular piece of legislation), you still need to give them an out of “No opinion.”
8. Use imbalanced rating scales. If you do use a scale, make it uneven. Your members need to be allowed a middle ground. If you give them an even scale, you’ve forced a response, either positive or negative. Allow for neutral feelings or you’ll skew your data.
9. Keep your surveys as brief as pos­sible. If it takes you or a staff member longer than 10 minutes to take it, it’s probably too long. Consider sending two shorter surveys stretched across a longer period of time. If you absolutely must send out a lengthier survey, make sure you’re offering your ­members an incentive to participate.

Survey Don’ts

1. Don’t make sensitive questions mandatory. Certain questions about income or transaction sides can make your respondents uneasy. Make those questions optional—and for income-related questions, put the responses in ranges. A member might not be willing to tell you that he made only $15,000 in real estate in 2014, but might be willing to say that he earned “less than $20,000.”
2. Don’t complicate the survey. If you need to give background before members can answer a question, make that background brief and to the point. No one will read a full page of anything before answering a question. And keep the questions themselves brief, clear, and conversational.

Survey vs Focus Group

Surveys (quantitative research) and focus groups (qualitative research) are not interchangeable. Which one to choose depends on your research objective or what you need to do with the information. If you need to explore or uncover issues or underlying reasons for behaviors, then you need a focus group. If you need to measure or determine behaviors or factors that you are already positive you understand, then you’re ready for a survey. If you aren’t even sure what your members’ greatest needs are right now or how to address them, you need to talk to some of them using qualitative research. You can’t measure how big the problems are if you’re not sure what the problems are in the first place.

Anonymous or Named?

Your surveys should always be anonymous. You can collect member information for a drawing at the end, but if you want honest responses, you need to assure your members that their individual responses will not be connected with any identifying information. Yet you can still ask demographic questions, such as function in real estate, age, years in real estate, gender, and income. These are good ways to segment your results to see if there are differences among groups. But your members need to know that that information will not be used to market to them and will not be passed on to a third party.