Tallying the success of the CEO's latest speech to stockholders poses much more of a problem than tallying sales.

"The guys in marketing can tell you how much Coca-Cola they've sold," says Robert Byrd, senior managing editor in executive, financial and employee communications for The Coca-Cola Company. "But our world is a little more nebulous."

It doesn't have to be. Most speechwriters use questionnaires to gauge whether an audience understood and responded to a speech. But the key to getting a true measurement is in the questionnaire's construction and content.

Keep them simple

Doug Garr, a speechwriter for IBM Senior V.P. Robert Stephenson, says he collects feedback on most major speeches using one-page comment forms.

The forms are very simple, so as not to dissuade audience members from responding. For instance, a form for a recent speech by an IBM exec at PC Expo asked listeners to rate the overall session, the content and the delivery (with ratings ranging from "very satisfied" to "very dissatisfied"). These were check-off, or "closed-end" questions; other questions on the form asked for written answers about "what I liked best" and "what I liked least."

"We have questionnaires that are specific to the audience," explains Garr. "If there are four or five speakers, we'll ask the audience to rate each one of them. . . . We pull no punches - if an executive stunk up the room, we'll tell him that."

Creating effective questionnaires

How can you go about setting up your own audience surveys for speech measurement? Here is a list of tips to help formulate an effective questionnaire:

• Don't waste time asking questions about the room's lighting or whether the CEO has a pleasant voice, says James Porchey, president of St. Louis-based market research firm Porchey Research Inc. "The most important thing is measuring whether the message is being communicated," he adds.
  
  
• The form should be short enough so that audience members can fill it out in five minutes or less. And it shouldn't be more than one side of a sheet of paper.
  
  
• Use questions that ask "what I liked most/least" to gauge whether the speech's message came through.
  
  
• Make sure the cards are distributed to the audience after the speech, Porchey advises. "You don't want them filling them out as they go," he says. "At the end of the speech, you can just have the moderator tell the audience that you'd like their help in evaluating what they've heard."
  
  
• Don't allow the speaker to hand out the evaluation forms, Porchey warns. Instead, let the moderator or some other neutral person do this task. Otherwise, the answers may be skewed.

Formatting conclusions

Tabulating the responses to closed-end questions is easy. But the open-ended, qualitative answers require a different approach. You'll need a trusted support person - or another speechwriter - to comb through the written responses and come to some conclusions. It's important that the person who wrote the speech doesn't compile the data. "You need a person who can make some objective observations," Porchey says.

Tip: The results should be distributed to speechwriters and speakers between one and two weeks after the speech itself. "Just like the audience forgets the speaker, the speechwriter and the speaker move on to different things after a few weeks," Porchey points out.

Often, the data confirms your own opinions of how the speech was received. But having the results provides ammunition for helping speakers change their speaking style, or perhaps craft a new message.

"If you send data back saying that someone was good or bad, they'll pay more attention to that," says Garr. "We use the results to show that we're not making this stuff up."

That kind of scrutiny, Garr adds, actually helps keep the speakers on their toes. "These guys are thinking a lot more about their speeches," he says. "They know they're going to be judged by an audience of customers - they can't just show off for the chairman."