Any rigorous research project takes time, energy, and good, old-fashioned elbow grease. Fortunately, I was able to convince several amazing people to help me complete the Middle Way Management Assessment Instrument™ (MWM-AI™) study. Special thanks go to Dr. Debra Elliott, James Rivera, and the inimitable Dr. Theodore Kariotis. These brave souls provided resources, comments, and encouragement throughout the process.
The Process
The process of completing a mixed-methods study is a lengthy one. In my case, it was composed of several steps:
- Delphi study to define and refine survey/questionnaire items,
- Limited field test of survey/questionnaire instrument - testing for validity, reliability, and internal consistency of items,
- Full-scale administration of validated survey/questionnaire instrument,
- Statistical analyses and write-up of full scale survey.
The Delphi Study
A Delphi study is simply a qualitative, panel-of-experts effort intended to create a reliable outcome based solely upon the expertise offered by the panel. In this case, the panel of experts was comprised of leaders and managers in several industry segments. This group included managers with years of experience at various educational levels, including undergraduate and graduate degrees.
A typical Delphi study allows the panel of experts to initially suggest several items for inclusion in a survey instrument. Since Middle Way Management™ is my own, new construct, I offered 100 items to the group along with an explanation of the guiding philosophy of Middle Way Management. Most of the panel participants had already been exposed to the approach via this blog or through personal communication. Regardless, the group completed two passes at the list of items. They eventually settled upon 25 items that I could take out to a limited field test.
Jim Rivera was instrumental in this phase of the study by pointing out that an executive manager's (e.g., C-level, President) point of view might be decidedly different than a front-line manager's. Based upon his input, I created a final instrument that captures the respondent's organizational level and branches to appropriate items. Thanks, Jim.
The Limited Field Test
Once the survey instrument had been defined and refined, I could embark upon the quantitative portion of the study. As a member of LinkedIn, I actively recruited several of my links to participate in the limited field study of the MWM-AI. I also recruited friends and colleagues, many of whom graciously participated.
In the end, 27 people completed the MWM-AI, a small sample, yet sufficient to test reliability using Cronbach's Alpha (this measurement essentially confirms that groups - in this case, all - of survey items measure "the same thing"). I was delighted to discover an alpha coefficient score of .964 over the 25 items of the instrument. Conventional social scientific wisdom dictates an acceptable level of > .80, though some researchers suggest > .70 is also acceptable. Either way, I was good to go - time to take it to a larger group.
The Real Deal
This is where Debra Elliott stepped up to the plate. Debra successfully negotiated with her organization to administer the final MWM-AI across all levels of her organization. Thanks, Debra. The results were interesting (I'm summarizing them in an article and won't bore you with the details here) and I was able to run Cronbach's Alpha on a larger sample of 137. The result: .966 across all 25 items. Again, good to go!
Future Directions
Now that I've validated the MWM-AI, it is truly ready to be delivered to the world. In fact, you can visit the survey here:
The Middle Way Management Assessment Instrument (MWM-AI)
Go ahead and punch away at it. I would be very interested in any comments or suggestions you may have to improve the instrument.
Now go, manage with compassion!
Onward! Darin
Resources that helped determine proper sample sizes for Cronbach's Alpha:
Duhachek, A., Coughlan, A. T., & Iacobucci, D. (2005, Spring). Results on the standard error of the coefficient alpha index of reliability. Marketing Science, 24(2), 294-301. doi:10.1287/mksc31040.0097
Duhachek, A., & Iacobucci, D. (2004). Alpha's standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 89(5), 792-808. doi:10.1037/0021-9010.89.5.792
Copyright © 2009/2010, Darin R. Molnar, PhD. All rights reserved.
No comments:
Post a Comment