More than 150 students, faculty and staff from KNUST and other higher education institutions in Africa gathered in person or virtually April 3 and 4 to learn about conducting experiments.
“Simple” is a word Mahyar Eftekhar uses often to describe experimental research. As he phrased it on a presentation slide, “The art of designing good experiments … is in creating simple environments that capture the essence of the real problem while abstracting away all unnecessary details.”
“The key is that if you want to design an experiment, you want to avoid creating a complicated environment,” said Eftekhar during his two-day faculty workshop in Ghana. “Again and again, I’m going to talk about this. Hopefully you will realize how simple you can design an experiment.”
Experiments can be conducted either in the lab or in the field. In lab experiments, researchers generate their own data and have more control.
“You invite subjects to come to your lab, wherever the lab is—a park, manufacturing lobby, a conference room, a classroom, wherever,” explained Eftekhar. “They know that they are contributing to an experiment. And then you record their behavior or you record their decisions. And this is the way that you generate data.”
Field experiments, on the other hand, take place in natural, real-world settings. The researcher has less control over what takes place.
“In the field experiment, you have some control, but everything is in nature,” Eftekhar said. “People don’t know that they contribute to your experiment. They show their real action, their real decision.”
Although the level of control is much higher in a lab experiment, the external validity is lower, explained Eftekhar. Because the lab environment is often artificial, study results may not be generalizable to other populations.
Whether you are doing a lab or a field experiment, you want to be sure to design a rigorous study so the results are useful for practitioners, Eftekhar noted. He listed three factors that make experimental research rigorous:
- Theoretical guidance—The hypotheses you develop should be based on existing assumptions or theories.
- Induced valuation—You need to have some reward for participants to test how they pursue their goals.
- Careful control of institutional structure—The strategic options and information you make available to participants should match the real world.
Lab Experiments
Lab experiments typically serve one of three major purposes, Eftekhar said. These are 1) to test and refine an existing theory; 2) to develop a new theory; or 3) to test new institutional designs.
So the first step in designing an experiment is to identify or develop the theory it will test, said Eftekhar. He noted that a good theory has three characteristics.
“It should be useful. It should be based on repetition. It should be real,” he said. “You know, it should be useful to help us learn about the world or to help us improve something.”
In one of his own lab experiments that he shared with the group, Eftekhar tested the theory that charity organizations reduce their donation income if they offer volunteering opportunities. He conducted the project in collaboration with a social service charity in Arizona, The Society of St. Vincent de Paul.
“This study is based on standard economic theory,” Eftekhar said. “People make one type of donation. Time is valuable. If I gave you time, I’m not going to give you my money too.”
He found that theory not to be true, however. In fact, in his study, volunteers were more likely to donate and at higher levels.
To run the experiment, Eftekhar recruited participants through an online research platform called Prolific. The study followed a between-subject design with two treatment conditions.
As Eftekhar noted, a study needs to include at least two treatments—a baseline and a comparison—to qualify as an experiment. A research project with only one treatment is considered a demonstration.
Participants were randomly assigned to either a volunteer or task group. Randomization is key in any experiment, Eftkhar stressed. It’s a way to avoid confounding the experiment with nuisance variables that throw off the results.
The volunteer group was asked to design a “sweet dreams” note card that would be left on shelter residents’ pillows at night. The task group was asked to design a card for themselves. They were told the purpose was to assess the artistic quality of virtual painting.
Each participant was paid $3 plus a $1.50 bonus. At the end, they were given the chance to donate all or any part of their bonus to St. Vincent de Paul. Participants who completed a volunteer activity were 15% more likely than the task group to make a donation, and their donations were 21% higher.
“This is a counterintuitive result compared to classic economic theories,” said Eftekhar. “In classic economic theories, we assume that if you’ve given your time, you won’t give money.
“But in this setting, we showed that no, you give time as a volunteer and you also give money. This understanding is critical when a charity schedules a volunteering program.”
Field Experiments
In the field experiment Eftekhar shared at the workshop, he was collaborating with the same charity. St. Vincent de Paul wanted to reduce the costs it was incurring from unwanted in-kind donations, such as stained clothing or damaged furniture.
The researchers were tasked with finding a way for the charity to discourage its donors from giving trash, while not discouraging them from making any future contributions. Again, simplicity was key to the experiment.
“If you tell the donors, ‘Hey, look, you are giving us trash,’ donors are not going to laugh,” said Eftekhar. “They are going to say, ‘Okay, I don’t give any kind of donation to you whatsoever.’
“So you need to nicely tell them, ‘Don’t give us trash.’ And the charity asked us to do this, and this became our experiment. You see, as simple as that.”
Eftekhar based the study on two behavioral interventions: social norms and information disclosure. Both are well-documented in research literature.
Social norms come into play when people match their behavior to acceptable group standards. Information disclosure refers to altering people’s behavior through information or education.
In Eftekhar’s experiment, he composed two emails to be sent to in-kind donors two days before the scheduled pickup of their items. Importantly, the researchers did not change anything else about the donation process.
One group (social norm) received an email that said the majority of donors give the charity items in very good condition that have a high likelihood of being sold. A second group (information disclosure) received an email stating that unsellable items cost the charity tens of thousands of dollars every month to dispose of. A third (baseline) group did not receive an email.
The researchers then trained pickup drivers to evaluate the quality of donated items and rate them on a scale of one to five. A rating of one meant the items were all garbage. A score of five indicated the items were all good.
In-kind donations from those who received and opened the social norm email earned an average rating of 3.33. Donations from the information disclosure group averaged a rating of only 2.69. The researchers found no significant difference between the information disclosure and baseline groups.
“The social norm message was really powerful,” said Eftekhar. “Now the charity sends the message with social norm to their donors.”
Before the experiment, St. Vincent de Paul received about 90 truckloads of “garbage” donations each month, Eftekhar reported. Now they receive fewer than 45 truckloads monthly.
Useful for humankind
Eftekhar concluded his presentation by urging participants to consider how their own research projects will make a difference, both to theory and practice.
“One important question for you to think about is, what is the goal of a business school researcher?” Eftekhar asked. “I think our goal is to improve the living standards of people.
“Basically, this is the goal of any scientist. And that’s why, if you work on theory, you want to develop a theory that is useful for humankind.”