Qualitative Coding: How to Analyze Open-Ended Responses

by Kelly J. Larson | January 2016

This document explains how to code open-ended responses (qualitative data) from surveys or other sources. To allow for meaningful and comparative analysis, each response should be reviewed and given a code or multiple codes based on similarities among the responses.

When referring to qualitative data, a code represents a word or short phrase that summarizes or attributes meaning to open-ended responses.[1]

The coding for each open-ended question should begin with “1” and continue for as many codes as needed to make sure the responses are appropriately considered when analyzing the data. For example, responses to the question “What do you like best about working at Questar?” might be organized with the following codes (or coding scheme):

  1. Coworkers
  2. My job
  3. Work environment
  4. Compensation
  5. Benefits
  6. Other

Once the responses are coded, the frequency of each code can be tallied and placed in a table, thereby allowing qualitative data to be measured and analyzed similarly to quantitative data.

Coding Process
The following steps outline the process of coding open-ended responses.

  • Step 1: Extract the open-ended responses from the raw data.
  • Step 2: Organize the responses in Excel. Each question and its responses should be in its own spreadsheet.
  • Step 3: Review each question’s responses and look for recurring comments and themes. Read each response carefully at least twice.
  • Step 4: Create a coding scheme for each question by listing the recurring responses and then numbering them, beginning with “1” up to a maximum number of reasonable codes.
  • Step 5: Apply the coding scheme to each response. Some responses may have multiple codes depending on how extensive and detailed the response is. Only code the most frequently mentioned responses. Code the least frequently mentioned responses as “other.”
  • Step 6: Have at least two other people read the coded responses to check for accuracy, consistency, and agreement among the reviewers.
  • Step 7: Create a filter in Excel to review the responses by code. This is helpful for reviewing the codes.
  • Step 8: Create code frequency pivot tables in Excel.

When applying the coding scheme, duplicate the responses that require more than one code. For example, a response to the question “What do you like best about working at Questar?” might be “my awesome coworkers and having a job I’m excited and passionate about.” If two of the codes from the coding scheme are 1. Coworkers and 2. My job, “my awesome coworkers” would be associated with one code and “having a job I’m excited and passionate about” would be associated with the other. Figure 1 shows what the response would look like in Excel.

figure 1

Once each response is given a code or multiple codes depending on the depth of the response, a filter can be applied to the Excel spreadsheet to review the responses by code, and pivot tables can be created to display the code frequencies and allow for comparative analysis of the open-ended responses. The following steps outline the process of creating a pivot table in Excel.

  • Step 1: Each open-ended question and its responses should be in its own Excel spreadsheet. The heading for the column containing the responses should be “Response” and the heading for the column containing the codes should be “Code,” as shown in Figure 2.
  • Step 2: Click on the “Insert” tab.
  • Step 3: Click on “PivotTable” on the far left.
  • Step 4: A “Create PivotTable” box will appear. Make sure all the responses and codes are included in the Table/Range (including the headings!). Place the pivot table in the existing worksheet. Click on a cell to determine the pivot table’s location within the spreadsheet. Click OK.
  • Step 5: A “PivotTable Fields” box will appear. Drag the “Code” field to the bottom left box labeled “Rows.”
  • Step 6: Drag the “Code” field to the bottom right box labeled “Values” to create the code frequencies. It might automatically appear as “Sum of Code,” but you must change that by clicking on it and then choosing “Value Field Settings.” Change “Sum” to “Count.” Click OK.
  • Step 7: Drag the “Code” field to the “Values” box again, except this time you will produce the code percentages. It might automatically appear as “Sum of Code,” but you must change that by clicking on it and then choosing “Value Field Settings.” Change “Sum” to “Count.” Click OK. Next, right click on any cell in the corresponding pivot table column labeled “Count of Code2.” Place your cursor above “Show Values As” and click on “% of Grand Total.” You can change the number of decimal places by right clicking on the column and choosing “Format Cells.” Click “Percentage” and change the decimal places to either one or two. Click OK.
  • Step 8: Change the table’s labeling from “Row Labels” to “Code,” from “Count of Code” to “Freq.,” and from “Count of Code2” to “%.” Format as needed.

Figure 2 presents a code frequency table in Word that was created using pivot tables in Excel.

figure 2

The next step after creating the code frequency tables is to summarize the results in a summary analysis report. Create charts when possible to display the results, include the code frequency tables in an appendix, and insert some responses in the body of the report as examples when presenting your findings.

[1] Saldaña, J. (2013). An introduction to codes and coding. In The coding manual for qualitative researchers (p. 3). Los Angeles: Sage Publications.

Library green roof discussion continues

by Kelly J. Larson | J335 | Spring 2011

An environmentally friendly rooftop that would include dirt and perennials might be added to the renovation plans for the Madison Central Library.

Madison residents will continue to provide input on the inclusion of this eco-rooftop, called an extensive green roof, at a Madison Public Library Board meeting on Thursday at 7 p.m. at the Madison Senior Center on 330 W. Mifflin St., main floor.

The Facilities and Sustainability Department conducted a Green Roof Feasibility Study to determine whether the current budget for the library renovations can accommodate the roof. The study compared two types of green roofs: a rooftop community garden that would support vegetable growth and an extensive green roof that would have less soil and plant life.

According to the study, a rooftop community garden would cost an estimated $775,344 and another $149,850 in annual maintenance fees, whereas an extensive green roof would cost an estimated $156,000.

Thus far, city officials have rejected plans for a community garden because of the high costs. The current anticipated cost for the total renovations is $29.5 million, and these costs would mean a budget decrease in other areas such as heating, cooling and insulation.

Green roofs have many advantages such as being visually pleasing, extending roof life and reducing water runoff, which is why an extensive green roof option remains on the renovation design. The architects expect to be done with plans by April, and they continue to welcome public comments and suggestions.

Survey Summary: Career Readiness Assessment a Viable Option for High School Students

by Kelly Larson and Drs. Joe Orban, Canda Mueller, and Tim Vansickle | Questar Assessment Brief | March 2015

A career readiness survey was sent to high school counselors from a Midwestern state in April 2013 and again in September 2013. The purpose of this survey was to determine the usefulness of an assessment, or assessment modules, that would offer detailed career guidance for high school students transitioning to areas other than four-year degree programs. These areas could include vocational technical schools, community colleges, and the workforce.

Seventy-five people responded to the survey, which asked about respondents’ current resources and what additional resources may help them when working with students to be career ready. Survey responses were anonymous and will be used to determine if and how a career readiness assessment would be helpful to students, teachers, and parents.

The education industry focuses much attention on determining whether students are prepared to be successful in college or in a career. However, while two well known and widely used college readiness programs are available (the ACT and SAT), very few definitive career readiness programs, products, or systems exists. Currently, the ACT WorkKeys® and career readiness certifications are addressing these student needs, but much more can be done given the statistics for post high school students.

When parents of high school seniors are asked about their student’s post high school plans, nearly 90 percent indicate their son or daughter is going to a four-year college. When those same students are surveyed, about 70 percent indicate they are going to college. In reality, approximately 50 percent of a high school cohort will actually continue on to higher education. Of those students who do attend, only 50 percent (i.e., 25 percent of a high school cohort) receive a four-year college degree, which typically takes four or five years to obtain. This leaves 75 percent of every high school graduating class needing to be career ready.[1]

Some high school students are not fully aware of their career interests or skills and cannot determine what their proper career path should be. For those going to a four-year college, this may not be a problem because of the extended time and degree options available to them, which expose them to a variety of career choices. But for those students whose interests and skills may be best suited for vocational technical schools or apprenticeships, the time to make a career decision may be limited. This survey will help Questar in its mission of providing an assessment that serves the needs of these individuals.

Questar sent the career readiness survey to high school counselors because these individuals are directly involved in assisting students in identifying career options. The survey questions were developed within Questar through research and a series of meetings to provide a succinct and comprehensive survey. The online survey was distributed to the respondents by the state director most involved with working with the counselors. Month-long survey windows were provided in the spring and again in the fall with one follow-up reminder before the end of the survey window.

While statistics were not available on the internal statewide distribution of the survey, it is estimated that about one-third of the potential high school counselors in the state responded to the survey.

Data Analysis
As Figure 1 shows, most respondents were counselors, followed by guidance directors. The remaining 7 percent of respondents coded as “other” included a high school business teacher, a college and career coordinator, an administrator, an assistant principal, and a guidance intern.

figure 1

Sixty percent of respondents had been in their current position for 10 years or less, 30 percent had been there between 11 and 20 years, and 10 percent had been at their current position for 21 years or more. The average number of years respondents had been at their current position was 10 years.

When asked if their current assessments adequately cover the academic skills needed for careers, 29 percent of respondents answered with “mostly yes” or “definitely yes,” whereas 71 percent responded that their assessments definitely don’t, mostly don’t, or somewhat but not adequately cover career-focused academic skills.

When asked if the current assessments adequately cover the workplace skills needed for careers, 21 percent of respondents answered “mostly yes” or “definitely yes,” whereas 79 percent responded that their assessments definitely don’t, mostly don’t, or somewhat but not adequately cover necessary workplace skills.

When asked if they struggle to find resources that are understandable for parents and students, 77 percent of respondents responded “yes” and 23 percent responded “no.” When asked how often students go to them for help in planning their careers, 49 percent of respondents replied “seldom” or “occasionally” and 51 percent responded with “often” or “always.”

As shown in Figure 2, most respondents thought students should start asking for career guidance as early as middle school and no later than their sophomore year in high school.

figure 2

As shown in Figure 3, most respondents thought guidance counselors and specific Internet websites are the best places for students to find career guidance, followed by parents and general Web searches. Suggestions from the 25 people who provided an “other” answer included job shadowing or hands-on experience, teachers and classrooms, career and college experts, and online resources.

figure 3

When asked what is the most time an assessment should take to measure workplace skills, more than 80 percent of respondents chose either 30 or 45 minutes, as shown in Figure 4.

figure 4

When asked which type of career assessment they believed students would prefer, most respondents (89 percent) chose the shorter assessment for immediate feedback, whereas 11 percent chose a longer assessment for more detailed but delayed feedback.

Respondents were posed the following hypothetical situation: “Suppose you were designing a score report for an assessment that measured workplace skills. Please indicate how useful each of the following would be to you, parents, and students.” As Figure 5 shows, most respondents thought that each of the 12 areas were mostly useful or definitely most useful.

In particular, 72 percent of respondents indicated that “Information on what careers would be most suitable for the skills assessed” would be “definitely most useful,” and 61 percent of respondents indicated that “Information on how to improve the workplace skills” would be “definitely most useful.” These two areas received the highest percentages for that answer. One person also suggested that local and regional jobs in suggested career fields or a link to sites of interest in those fields would be helpful.

figure 5

Of the 39 people who responded to the open-ended question that asked them to describe their thoughts about career assessment and how they would see it being used at their school, approximately 50 percent of them indicated that an assessment dedicated to career readiness would be important and useful.

They noted that this type of assessment would greatly benefit students who do not go on to a four-year college, especially since many of them often do not know what their skills are and lack direction in choosing a path after high school. Even those students who do go on to a four-year college would benefit from a career readiness assessment. Schools focus on students becoming college-ready, but, as respondents commented, it would be nice to have a tool geared toward helping students become career-ready as well. Respondents would use results from a career-focused assessment to provide students with more individualized and clearer direction.

Guidance counselors and directors seem to have a need for an assessment dedicated to career readiness. Even if they already have a career readiness process in place at their school, any tool geared toward helping students define a clearer post-graduation path is, for the most part, welcomed.

According to the response data from this survey, a career readiness assessment should be short and provide immediate feedback. It should ask students questions about their skills and traits such as integrity, ability to follow directions, attitude toward supervision, attitude toward coworkers, and workplace reading and comprehension. Reports should include information on what careers would be most suitable for the skills assessed, information on how to improve workplace skills, information on what workplace skills are strongest and weakest, vocational interest, information on how to improve workplace skills, an overall score for the workplace skills, and an overall score combining the academic and workplace skills.

Given the large number of skills and traits for which information is desired, the assessment could not be as short as respondents have indicated. However, significant reduction in assessment time can be achieved where computer adaptive testing (CAT) is applicable. This can be addressed by making a module or subtest to adhere to the respondents’ thoughts on time of administration. By building a career readiness assessment in modules, quick results and adequate time to administer can be achieved. Modules also allow both the student and the counselor to target areas that a student could explore more than others. For example, a student who is doing well in AP Calculus may not need to assess their mathematics readiness skills and could instead concentrate on the personal traits and skills portions of career readiness.

The responses in this survey provide an insider look at how those directly in contact with students view career readiness and the effectiveness of a career readiness assessment. Therefore, these responses are important to the designing and implementation of a career readiness tool. They are also important because the results highlight the need for a comprehensive instrument. Future research should include a larger sample of responses from multiple states.

t how those directly in contact with students view career readiness and the effectiveness of a career readiness assessment. Therefore, these responses are important to the designing and implementation of a career readiness tool. They are also important because the results highlight the need for a comprehensive instrument. Future research should include a larger sample of responses from multiple states.

[1] ACT. (2003). Relationships Between EPAS Scores and College Preparatory Course Work in High School, ACT Research Report Series. David J. Woodruff, ACT Research Division.

Texas Roadhouse: Fried Chicken vs. Ribs vs. Pork

by Kelly J. Larson | The Dish | May 2012

If you like your chicken fried and a cold beer on a Friday night, Texas Roadhouse may be the right place for you. With the radio up playing country music and peanut shells littering the floor, this restaurant offers “blue-ribbon” winning, fall-off-the-bone ribs and a slab of fried chicken that will surprise you.

“Literally, it’s the size of my face,” UW senior Amanda Axel said.

Known for its steaks, the chain restaurant started 19 years ago in Clarksville, Ind., and now has more than 300 locations in 44 states. Its location on Madison’s far east side requires a car or bus ride from campus, but it’s worth it.

Texas Roadhouse is not a place to watch the sunrise, but you might be able to see the love in a woman’s eyes, feel the touch of a precious child or know a mother’s love because it’s very family friendly. People of all ages can enjoy watching the staff dance to a song or a birthday guy or gal sit on the saddle and receive a happy birthday “yeehaw.”

Not wanting to miss out, I ventured east with my friends Amanda Axel and Jackie Nelson to determine whether Texas Roadhouse offers better fried chicken or “fall-off-the-bone” ribs.

When I asked the waiter why the ribs were dubbed “blue-ribbon” winning, he said, “I have no idea. I’ve never seen an award, but apparently they win every year.”

Regardless, they usually do fall off the bone as its name suggests. Amanda ordered the country fried chicken. It was breaded and very thin, and, according to Amanda, it tasted “like a giant chicken nugget.”

Jackie ordered the pulled pork.

We talked and laughed as we ate and drank. Dinner with friends is always a good time, and if you agree, have a drink with me and raise your glasses for a toast because the moment has come for me to make a decision: The ribs beat the chicken.

Amanda would disagree, and Jackie would vouch for the pork.

Despite our different opinions, we learned that whether you’re in the mood for fried chicken, ribs or pork, Texas Roadhouse is simply a fun place to go.

“I’d definitely come back,” Jackie said.

It really is the little things in life that mean the most, not where you live, what you drive or the price tag on your clothes. Spending a couple hours with friends, stuffing my face with the best honey buns in town and wearing jeans that fit just right make me happy. That’s all I can really ask for when going out to eat.

Chemo Ducks comfort young cancer patients

by Kelly J. Larson | J335 | Spring 2011

Chemo Ducks provide comfort and companionship to young children during cancer treatment. The stuffed animals have tubes identical to those used for chemotherapy, as well as a matching bandana and an activity book that explains what the child is going through. Family members, nurses, doctors and child life specialists can also use Chemo Ducks to explain the cancer treatment process more easily to the young patients.

The American Family Children’s Hospital at the University of Wisconsin-Madison does not have these readily available for patients, but one hospital employee wants to change that.

chemo duck

Regina Yocum with a Chemo Duck

As a child life specialist, Regina Yocum helps young patients and their families through challenging experiences with illnesses. For children in the oncology department, Yocum likes to use Chemo Ducks.

“When a Chemo Duck isn’t available, I use other stuffed animals to help explain to the kids what they’re going through,” she said. “But the ducks are so much better for that.”

Chemo Ducks are part of Gabe’s Chemo Duck Program. According to the program’s website, it began in 2004 by Lu Sipos for her cancer-stricken son, Gabe. When Gabe was 1 year old, Sipos developed an idea for her son to have a companion throughout his cancer experience and gave Gabe the first Chemo Duck.

One Chemo Duck costs $25, but they can be bought in a bulk of 12 for $250. Because the UW Children’s Hospital does not provide funding to purchase the ducks, Yocum and the families that want one have to purchase them on their own or rely on donations from the program’s website or other sources.

Yocum’s goal is to get the program to become part of the hospital’s budget: “I would love to be able to give one right away to a family who comes rushing into the hospital on a Thursday night instead of waiting a month to get one.”

Recently, Alpha Phi Omega, a volunteer service fraternity at UW-Madison, raised $275 for the UW Children’s Hospital to purchase the minimum 12 Chemo Ducks. This will allow Yocum and others at the hospital to distribute the stuffed animals more quickly to the young oncology patients in need of companionship and comfort as they endure cancer treatment.