Qualitative Coding: How to Analyze Open-Ended Responses

by Kelly J. Larson | January 2016

Introduction
This document explains how to code open-ended responses (qualitative data) from surveys or other sources. To allow for meaningful and comparative analysis, each response should be reviewed and given a code or multiple codes based on similarities among the responses.

When referring to qualitative data, a code represents a word or short phrase that summarizes or attributes meaning to open-ended responses.[1]

The coding for each open-ended question should begin with “1” and continue for as many codes as needed to make sure the responses are appropriately considered when analyzing the data. For example, responses to the question “What do you like best about working at Questar?” might be organized with the following codes (or coding scheme):

  1. Coworkers
  2. My job
  3. Work environment
  4. Compensation
  5. Benefits
  6. Other

Once the responses are coded, the frequency of each code can be tallied and placed in a table, thereby allowing qualitative data to be measured and analyzed similarly to quantitative data.

Coding Process
The following steps outline the process of coding open-ended responses.

  • Step 1: Extract the open-ended responses from the raw data.
  • Step 2: Organize the responses in Excel. Each question and its responses should be in its own spreadsheet.
  • Step 3: Review each question’s responses and look for recurring comments and themes. Read each response carefully at least twice.
  • Step 4: Create a coding scheme for each question by listing the recurring responses and then numbering them, beginning with “1” up to a maximum number of reasonable codes.
  • Step 5: Apply the coding scheme to each response. Some responses may have multiple codes depending on how extensive and detailed the response is. Only code the most frequently mentioned responses. Code the least frequently mentioned responses as “other.”
  • Step 6: Have at least two other people read the coded responses to check for accuracy, consistency, and agreement among the reviewers.
  • Step 7: Create a filter in Excel to review the responses by code. This is helpful for reviewing the codes.
  • Step 8: Create code frequency pivot tables in Excel.

When applying the coding scheme, duplicate the responses that require more than one code. For example, a response to the question “What do you like best about working at Questar?” might be “my awesome coworkers and having a job I’m excited and passionate about.” If two of the codes from the coding scheme are 1. Coworkers and 2. My job, “my awesome coworkers” would be associated with one code and “having a job I’m excited and passionate about” would be associated with the other. Figure 1 shows what the response would look like in Excel.

figure 1

Once each response is given a code or multiple codes depending on the depth of the response, a filter can be applied to the Excel spreadsheet to review the responses by code, and pivot tables can be created to display the code frequencies and allow for comparative analysis of the open-ended responses. The following steps outline the process of creating a pivot table in Excel.

  • Step 1: Each open-ended question and its responses should be in its own Excel spreadsheet. The heading for the column containing the responses should be “Response” and the heading for the column containing the codes should be “Code,” as shown in Figure 2.
  • Step 2: Click on the “Insert” tab.
  • Step 3: Click on “PivotTable” on the far left.
  • Step 4: A “Create PivotTable” box will appear. Make sure all the responses and codes are included in the Table/Range (including the headings!). Place the pivot table in the existing worksheet. Click on a cell to determine the pivot table’s location within the spreadsheet. Click OK.
  • Step 5: A “PivotTable Fields” box will appear. Drag the “Code” field to the bottom left box labeled “Rows.”
  • Step 6: Drag the “Code” field to the bottom right box labeled “Values” to create the code frequencies. It might automatically appear as “Sum of Code,” but you must change that by clicking on it and then choosing “Value Field Settings.” Change “Sum” to “Count.” Click OK.
  • Step 7: Drag the “Code” field to the “Values” box again, except this time you will produce the code percentages. It might automatically appear as “Sum of Code,” but you must change that by clicking on it and then choosing “Value Field Settings.” Change “Sum” to “Count.” Click OK. Next, right click on any cell in the corresponding pivot table column labeled “Count of Code2.” Place your cursor above “Show Values As” and click on “% of Grand Total.” You can change the number of decimal places by right clicking on the column and choosing “Format Cells.” Click “Percentage” and change the decimal places to either one or two. Click OK.
  • Step 8: Change the table’s labeling from “Row Labels” to “Code,” from “Count of Code” to “Freq.,” and from “Count of Code2” to “%.” Format as needed.

Figure 2 presents a code frequency table in Word that was created using pivot tables in Excel.

figure 2

The next step after creating the code frequency tables is to summarize the results in a summary analysis report. Create charts when possible to display the results, include the code frequency tables in an appendix, and insert some responses in the body of the report as examples when presenting your findings.

[1] Saldaña, J. (2013). An introduction to codes and coding. In The coding manual for qualitative researchers (p. 3). Los Angeles: Sage Publications.

Advertisements

Library green roof discussion continues

by Kelly J. Larson | J335 | Spring 2011

An environmentally friendly rooftop that would include dirt and perennials might be added to the renovation plans for the Madison Central Library.

Madison residents will continue to provide input on the inclusion of this eco-rooftop, called an extensive green roof, at a Madison Public Library Board meeting on Thursday at 7 p.m. at the Madison Senior Center on 330 W. Mifflin St., main floor.

The Facilities and Sustainability Department conducted a Green Roof Feasibility Study to determine whether the current budget for the library renovations can accommodate the roof. The study compared two types of green roofs: a rooftop community garden that would support vegetable growth and an extensive green roof that would have less soil and plant life.

According to the study, a rooftop community garden would cost an estimated $775,344 and another $149,850 in annual maintenance fees, whereas an extensive green roof would cost an estimated $156,000.

Thus far, city officials have rejected plans for a community garden because of the high costs. The current anticipated cost for the total renovations is $29.5 million, and these costs would mean a budget decrease in other areas such as heating, cooling and insulation.

Green roofs have many advantages such as being visually pleasing, extending roof life and reducing water runoff, which is why an extensive green roof option remains on the renovation design. The architects expect to be done with plans by April, and they continue to welcome public comments and suggestions.

Survey Summary: Career Readiness Assessment a Viable Option for High School Students

by Kelly Larson and Drs. Joe Orban, Canda Mueller, and Tim Vansickle | Questar Assessment Brief | March 2015

Introduction
A career readiness survey was sent to high school counselors from a Midwestern state in April 2013 and again in September 2013. The purpose of this survey was to determine the usefulness of an assessment, or assessment modules, that would offer detailed career guidance for high school students transitioning to areas other than four-year degree programs. These areas could include vocational technical schools, community colleges, and the workforce.

Seventy-five people responded to the survey, which asked about respondents’ current resources and what additional resources may help them when working with students to be career ready. Survey responses were anonymous and will be used to determine if and how a career readiness assessment would be helpful to students, teachers, and parents.

Background
The education industry focuses much attention on determining whether students are prepared to be successful in college or in a career. However, while two well known and widely used college readiness programs are available (the ACT and SAT), very few definitive career readiness programs, products, or systems exists. Currently, the ACT WorkKeys® and career readiness certifications are addressing these student needs, but much more can be done given the statistics for post high school students.

When parents of high school seniors are asked about their student’s post high school plans, nearly 90 percent indicate their son or daughter is going to a four-year college. When those same students are surveyed, about 70 percent indicate they are going to college. In reality, approximately 50 percent of a high school cohort will actually continue on to higher education. Of those students who do attend, only 50 percent (i.e., 25 percent of a high school cohort) receive a four-year college degree, which typically takes four or five years to obtain. This leaves 75 percent of every high school graduating class needing to be career ready.[1]

Some high school students are not fully aware of their career interests or skills and cannot determine what their proper career path should be. For those going to a four-year college, this may not be a problem because of the extended time and degree options available to them, which expose them to a variety of career choices. But for those students whose interests and skills may be best suited for vocational technical schools or apprenticeships, the time to make a career decision may be limited. This survey will help Questar in its mission of providing an assessment that serves the needs of these individuals.

Method
Questar sent the career readiness survey to high school counselors because these individuals are directly involved in assisting students in identifying career options. The survey questions were developed within Questar through research and a series of meetings to provide a succinct and comprehensive survey. The online survey was distributed to the respondents by the state director most involved with working with the counselors. Month-long survey windows were provided in the spring and again in the fall with one follow-up reminder before the end of the survey window.

While statistics were not available on the internal statewide distribution of the survey, it is estimated that about one-third of the potential high school counselors in the state responded to the survey.

Data Analysis
As Figure 1 shows, most respondents were counselors, followed by guidance directors. The remaining 7 percent of respondents coded as “other” included a high school business teacher, a college and career coordinator, an administrator, an assistant principal, and a guidance intern.

figure 1

Sixty percent of respondents had been in their current position for 10 years or less, 30 percent had been there between 11 and 20 years, and 10 percent had been at their current position for 21 years or more. The average number of years respondents had been at their current position was 10 years.

When asked if their current assessments adequately cover the academic skills needed for careers, 29 percent of respondents answered with “mostly yes” or “definitely yes,” whereas 71 percent responded that their assessments definitely don’t, mostly don’t, or somewhat but not adequately cover career-focused academic skills.

When asked if the current assessments adequately cover the workplace skills needed for careers, 21 percent of respondents answered “mostly yes” or “definitely yes,” whereas 79 percent responded that their assessments definitely don’t, mostly don’t, or somewhat but not adequately cover necessary workplace skills.

When asked if they struggle to find resources that are understandable for parents and students, 77 percent of respondents responded “yes” and 23 percent responded “no.” When asked how often students go to them for help in planning their careers, 49 percent of respondents replied “seldom” or “occasionally” and 51 percent responded with “often” or “always.”

As shown in Figure 2, most respondents thought students should start asking for career guidance as early as middle school and no later than their sophomore year in high school.

figure 2

As shown in Figure 3, most respondents thought guidance counselors and specific Internet websites are the best places for students to find career guidance, followed by parents and general Web searches. Suggestions from the 25 people who provided an “other” answer included job shadowing or hands-on experience, teachers and classrooms, career and college experts, and online resources.

figure 3

When asked what is the most time an assessment should take to measure workplace skills, more than 80 percent of respondents chose either 30 or 45 minutes, as shown in Figure 4.

figure 4

When asked which type of career assessment they believed students would prefer, most respondents (89 percent) chose the shorter assessment for immediate feedback, whereas 11 percent chose a longer assessment for more detailed but delayed feedback.

Respondents were posed the following hypothetical situation: “Suppose you were designing a score report for an assessment that measured workplace skills. Please indicate how useful each of the following would be to you, parents, and students.” As Figure 5 shows, most respondents thought that each of the 12 areas were mostly useful or definitely most useful.

In particular, 72 percent of respondents indicated that “Information on what careers would be most suitable for the skills assessed” would be “definitely most useful,” and 61 percent of respondents indicated that “Information on how to improve the workplace skills” would be “definitely most useful.” These two areas received the highest percentages for that answer. One person also suggested that local and regional jobs in suggested career fields or a link to sites of interest in those fields would be helpful.

figure 5

Of the 39 people who responded to the open-ended question that asked them to describe their thoughts about career assessment and how they would see it being used at their school, approximately 50 percent of them indicated that an assessment dedicated to career readiness would be important and useful.

They noted that this type of assessment would greatly benefit students who do not go on to a four-year college, especially since many of them often do not know what their skills are and lack direction in choosing a path after high school. Even those students who do go on to a four-year college would benefit from a career readiness assessment. Schools focus on students becoming college-ready, but, as respondents commented, it would be nice to have a tool geared toward helping students become career-ready as well. Respondents would use results from a career-focused assessment to provide students with more individualized and clearer direction.

Conclusion
Guidance counselors and directors seem to have a need for an assessment dedicated to career readiness. Even if they already have a career readiness process in place at their school, any tool geared toward helping students define a clearer post-graduation path is, for the most part, welcomed.

According to the response data from this survey, a career readiness assessment should be short and provide immediate feedback. It should ask students questions about their skills and traits such as integrity, ability to follow directions, attitude toward supervision, attitude toward coworkers, and workplace reading and comprehension. Reports should include information on what careers would be most suitable for the skills assessed, information on how to improve workplace skills, information on what workplace skills are strongest and weakest, vocational interest, information on how to improve workplace skills, an overall score for the workplace skills, and an overall score combining the academic and workplace skills.

Given the large number of skills and traits for which information is desired, the assessment could not be as short as respondents have indicated. However, significant reduction in assessment time can be achieved where computer adaptive testing (CAT) is applicable. This can be addressed by making a module or subtest to adhere to the respondents’ thoughts on time of administration. By building a career readiness assessment in modules, quick results and adequate time to administer can be achieved. Modules also allow both the student and the counselor to target areas that a student could explore more than others. For example, a student who is doing well in AP Calculus may not need to assess their mathematics readiness skills and could instead concentrate on the personal traits and skills portions of career readiness.

The responses in this survey provide an insider look at how those directly in contact with students view career readiness and the effectiveness of a career readiness assessment. Therefore, these responses are important to the designing and implementation of a career readiness tool. They are also important because the results highlight the need for a comprehensive instrument. Future research should include a larger sample of responses from multiple states.

t how those directly in contact with students view career readiness and the effectiveness of a career readiness assessment. Therefore, these responses are important to the designing and implementation of a career readiness tool. They are also important because the results highlight the need for a comprehensive instrument. Future research should include a larger sample of responses from multiple states.

[1] ACT. (2003). Relationships Between EPAS Scores and College Preparatory Course Work in High School, ACT Research Report Series. David J. Woodruff, ACT Research Division.

Texas Roadhouse: Fried Chicken vs. Ribs vs. Pork

by Kelly J. Larson | The Dish | May 2012

If you like your chicken fried and a cold beer on a Friday night, Texas Roadhouse may be the right place for you. With the radio up playing country music and peanut shells littering the floor, this restaurant offers “blue-ribbon” winning, fall-off-the-bone ribs and a slab of fried chicken that will surprise you.

“Literally, it’s the size of my face,” UW senior Amanda Axel said.

Known for its steaks, the chain restaurant started 19 years ago in Clarksville, Ind., and now has more than 300 locations in 44 states. Its location on Madison’s far east side requires a car or bus ride from campus, but it’s worth it.

Texas Roadhouse is not a place to watch the sunrise, but you might be able to see the love in a woman’s eyes, feel the touch of a precious child or know a mother’s love because it’s very family friendly. People of all ages can enjoy watching the staff dance to a song or a birthday guy or gal sit on the saddle and receive a happy birthday “yeehaw.”

Not wanting to miss out, I ventured east with my friends Amanda Axel and Jackie Nelson to determine whether Texas Roadhouse offers better fried chicken or “fall-off-the-bone” ribs.

When I asked the waiter why the ribs were dubbed “blue-ribbon” winning, he said, “I have no idea. I’ve never seen an award, but apparently they win every year.”

Regardless, they usually do fall off the bone as its name suggests. Amanda ordered the country fried chicken. It was breaded and very thin, and, according to Amanda, it tasted “like a giant chicken nugget.”

Jackie ordered the pulled pork.

We talked and laughed as we ate and drank. Dinner with friends is always a good time, and if you agree, have a drink with me and raise your glasses for a toast because the moment has come for me to make a decision: The ribs beat the chicken.

Amanda would disagree, and Jackie would vouch for the pork.

Despite our different opinions, we learned that whether you’re in the mood for fried chicken, ribs or pork, Texas Roadhouse is simply a fun place to go.

“I’d definitely come back,” Jackie said.

It really is the little things in life that mean the most, not where you live, what you drive or the price tag on your clothes. Spending a couple hours with friends, stuffing my face with the best honey buns in town and wearing jeans that fit just right make me happy. That’s all I can really ask for when going out to eat.

Chemo Ducks comfort young cancer patients

by Kelly J. Larson | J335 | Spring 2011

Chemo Ducks provide comfort and companionship to young children during cancer treatment. The stuffed animals have tubes identical to those used for chemotherapy, as well as a matching bandana and an activity book that explains what the child is going through. Family members, nurses, doctors and child life specialists can also use Chemo Ducks to explain the cancer treatment process more easily to the young patients.

The American Family Children’s Hospital at the University of Wisconsin-Madison does not have these readily available for patients, but one hospital employee wants to change that.

chemo duck

Regina Yocum with a Chemo Duck

As a child life specialist, Regina Yocum helps young patients and their families through challenging experiences with illnesses. For children in the oncology department, Yocum likes to use Chemo Ducks.

“When a Chemo Duck isn’t available, I use other stuffed animals to help explain to the kids what they’re going through,” she said. “But the ducks are so much better for that.”

Chemo Ducks are part of Gabe’s Chemo Duck Program. According to the program’s website, it began in 2004 by Lu Sipos for her cancer-stricken son, Gabe. When Gabe was 1 year old, Sipos developed an idea for her son to have a companion throughout his cancer experience and gave Gabe the first Chemo Duck.

One Chemo Duck costs $25, but they can be bought in a bulk of 12 for $250. Because the UW Children’s Hospital does not provide funding to purchase the ducks, Yocum and the families that want one have to purchase them on their own or rely on donations from the program’s website or other sources.

Yocum’s goal is to get the program to become part of the hospital’s budget: “I would love to be able to give one right away to a family who comes rushing into the hospital on a Thursday night instead of waiting a month to get one.”

Recently, Alpha Phi Omega, a volunteer service fraternity at UW-Madison, raised $275 for the UW Children’s Hospital to purchase the minimum 12 Chemo Ducks. This will allow Yocum and others at the hospital to distribute the stuffed animals more quickly to the young oncology patients in need of companionship and comfort as they endure cancer treatment.

East Farmington intersection improvement project at a standstill

by Kelly J. Larson | The Osceola Sun | January 2014

IMG_4428

The project to improve the safety of the intersection at 30th Ave. and 280th St. in East Farmington has come to a halt. According to the town board at the Feb. 3 meeting, the original plan to lower the hill needs to be altered in order to do the job correctly. This has raised concerns of affordability and liability, and the board must now determine if it can, and should, walk away.

“We want to lower the hill down further,” Dennis Cottor, chairman of the board, said. “Right now you’re going to crest the hill and come down onto 280th. We believe as a board that we didn’t solve the problem. We improved on it but we didn’t solve it. And if we’re not going to solve it, maybe it’s best left alone.”

The town has already been approved for a $102,000 grant from the state of Wisconsin and would have spent $106,000 of its own budget on the original project. Altering the plan will cost the town an estimated $8,000 more in engineering expenses, plus additional construction costs.

“If we’re going to change this to do it right, I’m estimating $30,000–$40,000 more,” Cottor said.

Safety is the main concern at this intersection. Cars going in the ditch and “near misses” prompted the board to take action in the first place. If the board decides to discontinue the plan to modify the intersection because the cost has become too high, liability is a concern since the state has been made aware of the issue with the grant approval.

“We’re going to have another opinion of this done to find out where Farmington’s liability stands on this intersection,” Cottor said. “If we don’t have any liability to this, the consensus of the board is to walk away.”

Community members weighed in on the discussion at the meeting. Suggestions to improve the intersection without implementing the board’s construction project included widening the shoulders, increasing signage and placing guardrails.

The board will make a decision regarding the continuation of the construction project at the next board meeting.

Other items discussed at the Feb. 3 meeting included offering a bid for a new mower for the town, rental rates for equipment and the possibility of issuing a mailbox ordinance on new building permits to avoid snow removal problems.

The next board meeting is March 3 at 7 p.m. at the Town of Farmington town hall.

The Stepford Neighborhood

by Kelly J. Larson | J401: In-Depth Reporting | May 2012

More than 100 identical mailboxes line the streets of the Sauk Creek Neighborhood in Madison, Wis. Black geese, red foxes, red cardinals or golfers appear on cream boxes that sit atop simple wooden posts. Some look old and dirty while others are new and shiny.

Some stand out, though. Interspersed among the matching mailboxes include  black ones, gray ones, white ones and one that has flowers covering it. If someone were to complain about these deviant mailboxes, the neighborhood association could force the owners of the households to either trade them in for a cream-colored one or pay a fine.

“There are some people in the neighborhood who feel very strongly that they all need to match, and that if you replace your mailbox you have to get the geese,” Trish Ballweg, a Sauk Creek resident, said. “It’s like $150 for this mailbox, so there are others of us who feel that they’re really ugly and out of date, and who really cares if your mailbox matches as long as it looks nice?”

Sauk Creek, like many other neighborhoods across the country, has restrictive covenants. These rules, laid out by the developers of the neighborhood by contract, are meant to keep property values high by keeping the streets, yards and houses looking relatively similar and clear of features such as trailers parked in yards or firewood stacked against houses.

Other regulations in an official “declaration of covenants, restrictions and conditions” for Sauk Creek include the following: All single-family houses must have an attached garage with no less than two and no more than four stalls; households can have no more than two domestic animals; and only the mailboxes approved by the developer or the Architectural Control Committee, which is assigned by the developer to help regulate the restrictions, can be used.

In addition, any remodeling to the exterior of houses and any other changes that might alter the neighborhood’s appearance must first be approved by the architectural committee.

While these restrictions exist, Ballweg does not see them being a major issue: “I don’t think the neighborhood is obscenely strict about it. It would take somebody complaining about something for the [neighborhood association] board to do anything about it,” she said.

But complaints about neighbors do occur. Bill Schultz, president of the Sauk Creek Neighborhood Association, gave an example of a new resident who had a trailer parked in the yard. A neighbor called Schultz and complained about it.

“When a new person comes into the neighborhood, in part it’s up to the realtor to make sure they have that covenant,” Schultz said. “In this case the realtor never gave it to him, so we had to go and talk to them. They realized the situation and moved the trailer.”

A problem that Schultz notices is that some residents do not know the covenants even exist.

“The problem we’re having now is that [the covenant] is typically attached to a mortgage that is signed at closing when you buy a house.” he said. “Recently we’re finding out that the realtors, unintentionally or not, are not giving it to new buyers.”

The covenants are legally binding, and homebuyers who purchase a house within the covenant’s boundaries must adhere to them whether they know about the rules or not. If the neighborhood association comes across an issue where a resident has violated the covenant and refuses to make the necessary changes, it can give the resident a fine and take them to court if the fine is not paid voluntarily.

Professor Gretchen Viney at the UW-Madison Law School specializes in real estate law and said these covenants are becoming more and more common, especially in newer developments. People can do nothing about the rules and have to adhere to them, unless they can get 65 percent of the other households to vote to make a change.

“The due process concerns are quite telling,” Viney said. “There are a lot of rules about how government can govern its people. But in these homeowners associations, it’s not a government, it’s a private association. And so the rules that they can enforce can be a little bit frightening, from a freedoms point of view.”

Some people wanting to buy a house find the restrictions are not for them and continue to look elsewhere. Others are not bothered and find the restrictions add to the look and safety of the neighborhood.

Even if Sauk Creek residents disagree with the restrictions, for the most part they have to follow them to avoid conflict with neighbors and a possible fine. But having matching mailboxes seems to be one restriction people have been able to avoid, perhaps because the older, dirtier ones can damage Sauk Creek’s image.

“What happens is because they’re so expensive, people leave them up even though they need to be replaced – they start to look really awful,” Ballweg said. “What’s worse: if they put up a plain black one or leave the nasty one?”

The architectural committee sent a letter to each resident three years ago notifying them that the board was considering filing a lawsuit to force people to conform to the mailboxes. Schultz said it did not go over well, and it has not been much of an issue since.

“I think the board felt like there were bigger fish to fry,” he said.

Schultz is among those with a deviant mailbox. He bought one painted with flowers as a gift for his wife on Mother’s Day, not knowing the covenant required matching geese or birds.

“Would I ever change my mailbox? I’ve been thinking about it since I’m president of the neighborhood association,” Schultz said. “[But] my wife really likes that mailbox.”

The Sauk Creek Neighborhood Association said it wants to let residents know where they can go to get one of the mailboxes outlined in the covenant, as well as offer cleaning tips. The cream-colored ones are, after all, legally binding.