Analyzing Qualitative Data

🎯 Thesis Seminar-Module 12: Method VI

Making Sense of Stories: Analyzing Qualitative Data in ELT Teacher Training

Assessment, attendance, and calendar

Review assessment in Thesis Seminar Syllabus 2025.pdf, attendance according to Thesis Seminar Course Guidelines 2025.pdf, and Thesis Seminar 2025 Calendar.pdf.

Triangulate your data.

Triangulation in qualitative research means using multiple data sources, methods, investigators, or theories to study the same phenomenon. Its purpose is to increase the credibility, validity, and trustworthiness of your findings by cross-verifying information and capturing a more complete, multi-faceted picture. By comparing insights from different angles (like documents, interviews, and observations), you can build a more robust and convincing analysis.

Triangulating documents, interviews, observations, and questionnaires:

This specific combination is powerful because it allows you to compare:

• What people say they do or believe (Interviews)

• What they actually do in practice (Observations)

• What is formally planned, recorded, or reflected upon (Documents)

Here are several examples showing how you can triangulate these three sources in an ELT teacher training context:

💡

When reviewing the examples below, compare what you have and what research questions you are investigating to determine how your data (answers) still align with your original research questions. It’s common to have to “tweak” or to modify slightly your research questions to match the answers that you are pursuing through your data analysis. Modifying your research questions at this point is fine as long as they still relate to your literature review.

• Example 1: Evaluating Trainee Implementation of Communicative Language Teaching (CLT) Principles

  • Research Question: To what extent do trainees understand and implement CLT principles in their teaching after a methodology module?

  • Documents:

    • Analyze trainee lesson plans: Do the plans include communicative objectives? Do they feature pair/group work activities? Is the focus on meaning rather than just form?

    • Analyze trainee written reflections after teaching: Do they discuss successes/challenges related to student communication? Do they mention CLT principles?

  • Interviews:

    • Conduct semi-structured interviews with trainees: Ask them to define CLT in their own words. Ask how they plan activities to maximize student interaction. Explore their perceived challenges in implementing CLT.

  • Observations:

    • Conduct classroom observations: Does the observed lesson reflect the CLT features noted in the lesson plan? How much time do students spend communicating versus listening to the teacher? Does the teacher set up pair/group work effectively? Is the focus predominantly on accuracy or fluency?

  • How to Triangulate:

    • Compare the lesson plan (document) with the actual teaching (observation). Did the trainee implement the communicative activities as planned?

    • Compare the trainee’s stated understanding of CLT (interview) with their ability to design CLT activities (document - lesson plan) and implement them (observation).

    • Compare the trainee’s reflection on the lesson (document) with the observer’s notes (observation) and the trainee’s perspective shared in the interview. Do they highlight the same successes or challenges?

    • Finding: Triangulation might reveal a trainee who talks confidently about CLT principles (interview) and writes decent plans (document), but struggles with managing student interaction in practice (observation). This provides a much richer insight than any single source alone.

• Example 2: Investigating How Trainees Give Instructions for Activities

  • Research Question: How clear are trainees’ instructions, and how does this align with their planning and self-perception?

  • Documents:

    • Examine the “Procedure” section of lesson plans: Are instructions scripted or outlined? Do they include steps like using modeling or Instruction Checking Questions (ICQs)?

  • Interviews:

    • Interview trainees about their process for planning and delivering instructions. Ask what they consider important for clarity. Ask them to self-assess their strengths and weaknesses in giving instructions.

  • Observations:

    • Observe lessons, paying close attention to instruction-giving moments: Are instructions clear and concise? Does the trainee use visuals, modeling, or gestures? Do they check for understanding (ICQs)? How do students respond – do they seem to understand the task?

  • How to Triangulate:

    • Compare the planned instructions (document) with the delivered instructions (observation). Were planned ICQs actually used?

    • Compare the trainee’s self-assessment (interview) of their instruction-giving skills with their actual performance (observation).

    • Analyze if the student response (observation) indicates understanding, regardless of what the plan or interview suggested about clarity.

    • Finding: You might find a trainee whose lesson plan includes ICQs (document), but who forgets to use them in the moment (observation), and in the interview expresses general confidence but hasn’t identified this specific gap.

• Example 3: Understanding Trainee Beliefs and Practices Regarding L1 Use in the Classroom

  • Research Question: What are trainees’ beliefs about using the students’ first language (L1) in the English classroom, and how does this manifest in their teaching?

  • Documents:

    • Analyze teaching philosophy statements or essays where trainees might discuss their stance on L1 use.

    • Look at lesson plans: Do they explicitly plan for strategic L1 use (e.g., for explaining complex grammar, checking understanding) or aim for English-only?

  • Interviews:

    • Interview trainees directly about their views on L1 use: When is it acceptable? When should it be avoided? Why? How do their views align with program guidance?

  • Observations:

    • Observe lessons: Note instances where the trainee uses L1, and instances where students use L1. For what purposes is L1 used? How does the trainee respond to student L1 use? Does the observed practice align with their stated beliefs and plans?

  • How to Triangulate:

    • Compare the stated beliefs (interview, philosophy statement-document) with the planned approach (lesson plan-document) and the actual classroom behavior (observation).

    • Look for consistency or contradictions. A trainee might advocate for English-only (interview/document) but frequently resort to L1 when facing difficulties (observation).

    • Analyze why discrepancies might exist based on interview data (e.g., theoretical belief vs. practical pressure).

• Example 4: Assessing Trainee Engagement with Feedback

  • Research Question: How do trainees perceive and utilize the feedback provided by their supervisors?

  • Documents:

    • Analyze the written feedback provided on observation forms or lesson plans by the supervisor. What points were raised? What suggestions were made?

    • Look at the trainee’s subsequent lesson plan or written reflection: Is there evidence they attempted to address the feedback points?

  • Interviews:

    • Interview the trainee after they receive feedback: Ask them to summarize the feedback in their own words. What did they find most useful or surprising? How do they plan to act on it? Did they disagree with any points?

  • Observations:

    • Observe the feedback conference itself (if possible): How is feedback delivered and received? What is the nature of the dialogue?

    • Observe a subsequent lesson: Does the trainee’s teaching show evidence of incorporating the feedback suggestions discussed?

  • How to Triangulate:

    • Compare the supervisor’s written feedback (document) with the trainee’s interpretation of it (interview). Are they aligned?

    • Compare the trainee’s stated plan to act on feedback (interview) with evidence of action in subsequent teaching (observation) or planning (document).

    • Compare the tone/content of the written feedback (document) with the dynamics of the feedback conversation (observation).

By systematically comparing the insights gathered from documents, interviews, and observations, you move beyond relying on a single perspective. This process of triangulation allows you to report findings that are more nuanced, credible, and grounded in multiple forms of evidence.

What is coding?

Let’s delve into the world of qualitative data coding, specifically tailored for English language teacher trainers like yourself who are embarking on classroom-based research in applied linguistics or second language acquisition (SLA). Think of coding not as some arcane academic ritual, but as a systematic way to make sense of the rich, complex, and often messy reality of language teaching and learning as it happens.

What is Qualitative Data in Our Context?

First, what kind of data are we even talking about? In ELT research, qualitative data is non-numerical information that captures experiences, perspectives, processes, and meanings. This could include:

• Interview Transcripts: Conversations with teachers about their beliefs, practices, or challenges; discussions with learners about their strategies, motivations, or difficulties.

• Classroom Observation Field Notes: Your detailed descriptions of what happened during a lesson – teacher actions, student interactions, use of materials, classroom atmosphere.

• Audio/Video Recordings of Lessons: Capturing the dynamic flow of interaction, pronunciation features, use of L1/L2, etc.

• Student Work Samples: Essays, presentations, journal entries, online forum posts, visuals (photos, images, drawings, etc.), completed tasks that reveal language use, thought processes, or attitudes.

• Teacher Reflective Journals: Teachers’ own accounts of their teaching, decision-making, and reflections on lessons.

• Focus Group Discussions: Group interviews capturing shared perspectives or debates among students or teachers.

• School policies and lesson planning: lesson plans, scope and sequences, syllabus, curriculum, mission and vision statements, objectives, list of values, etc.

This data is rich, detailed, and often voluminous. You can’t just stare at 50 pages of interview transcripts and magically understand the key patterns. That’s where coding comes in.

• What is Qualitative Coding? The Core Idea

  At its heart, qualitative coding is the process of systematically identifying, labeling, and organizing segments of your data to discover patterns, themes, concepts, and relationships relevant to your research questions.

  Think of it like organizing a large, unsorted collection of photos from various school events. Initially, it’s just a pile. Coding is like:

  1. Looking closely at each photo (reading/listening to your data).

  2. Attaching labels (codes) to each photo based on what it shows (e.g., “Award Ceremony,” “Classroom Activity,” “Student Presentation,” “Teacher-Student Interaction,” “Informal Conversation”).

  3. Grouping photos with the same label together (gathering all data segments with the same code).

  4. Looking at the groups to see what larger stories or themes emerge (e.g., “Student Engagement Strategies,” “Challenges in Mixed-Ability Classes,” “Impact of Technology”).

  Essentially, you’re breaking down your data into smaller, manageable chunks and assigning a meaningful shorthand label (the code) to each chunk. This allows you to move from specific instances in the data to broader analytical insights.

• The Coding Process: An Iterative Journey

  Coding isn’t usually a neat, linear process. It’s often iterative and cyclical, meaning you’ll revisit your data and codes multiple times.8 Here’s a common flow:

  1. Immersion and Familiarization: Before you even think about codes, immerse yourself in the data. Read your transcripts multiple times, listen to recordings, review your field notes thoroughly. Get a holistic feel for the content. What jumps out at you initially?

  2. Initial / Open Coding: This is your first pass at labeling. Read through the data line-by-line or segment-by-segment (a segment could be a phrase, sentence, or paragraph) and assign codes that capture the essence of what’s being said or done.

     • Be descriptive: At this stage, codes can be very close to the data itself.

     • Use in vivo codes: Sometimes, the exact words used by participants are powerful and make excellent codes (e.g., a student saying “My brain just freezes” could be coded as brain freezes).

     • Don’t worry about perfection: Generate as many codes as seem relevant. You’ll refine them later.

     • Example (Teacher Interview Snippet): “Well, when a student makes a grammar mistake during a fluency activity, I usually don’t jump in immediately. I want them to keep talking, you know? Maybe I’ll subtly rephrase what they said correctly, or I might note it down to address later with the whole class if it’s common.”

     • Initial Codes might include: grammar mistake, fluency activity, delayed correction, prioritizing fluency, subtle rephrasing (or recast), noting errors, whole class feedback.

  3. Developing a Codebook (Crucial Step!): As you code, start creating a codebook or code list. This is a central document that lists:

     • Code Name: The label itself (e.g., Corrective Feedback Strategy).

     • Definition: A clear explanation of what the code means and represents.

     • Inclusion Criteria: When should you apply this code?

     • Exclusion Criteria: When shouldn’t you apply this code? (Helps distinguish similar codes).

     • Example Snippet: A quote from your data that exemplifies this code.

     This codebook ensures consistency, especially if multiple researchers are involved or if you return to the data later. It evolves as your understanding deepens.

  4. Focused / Axial Coding (Refining and Grouping): After the initial pass, review your codes.

     • Combine: Are some codes very similar? Can they be merged under a more conceptual label? (e.g., subtle rephrasing and implicit correction might become Implicit Corrective Feedback).

     • Split: Is a code too broad? Does it need to be broken down into more specific sub-codes? (e.g., Corrective Feedback Strategy might split into Implicit CF, Explicit CF, Metalinguistic CF).

     • Group: Start looking for relationships between codes. How do they connect? Which codes seem to belong together under larger categories or concepts? (e.g., Implicit CF, Explicit CF, Peer Correction, Self-Correction might all fall under the category Types of Error Treatment).

     • Example (Classroom Observation Snippet):

       • Field Note: “T asked students to discuss weekend plans in pairs. S1: ‘I go cinema.’ T nearby, smiled & said ‘Oh, you went to the cinema?’ S1 nodded, ‘Yes, went cinema.’ T moved on.”

       • Initial Codes: Pair work, Past tense error, Teacher proximity, Implicit Correction (Recast), Student Uptake (Partial).

       • Focused Coding: This instance clearly fits the refined code Implicit Corrective Feedback and perhaps Partial Learner Uptake.

  5. Identifying Themes / Selective Coding (Building the Narrative): This is where you move towards answering your research questions. You look at your refined codes and categories and identify the overarching themes, patterns, or narratives emerging from the data.

     • What are the most significant or frequent codes/categories?

     • How do different categories relate to each other? (e.g., Is there a relationship between Teacher Beliefs about Errors and the frequency of Implicit Corrective Feedback observed?)

     • What story does the data tell about the phenomenon you’re studying?

     • Example Theme: Based on codes like prioritizing fluency, delayed correction, Implicit CF, Fear of Demotivation, Learner Anxiety, a theme might emerge: “Teachers prioritize maintaining communicative flow and reducing learner anxiety over immediate grammatical accuracy during fluency-focused activities.”

Types of Codes with ELT Examples:

• Descriptive Codes: Basic labels for what is happening.

  • Teacher Question

  • Student Response (L1)

  • Using Visual Aid

  • Pair Work Instruction

  • Pronunciation Drill

• Topic Codes: Summarize the main topic being discussed or observed.

  • Vocabulary Learning Strategies

  • Challenges with Listening Comprehension

  • Motivation in Online Learning

  • Classroom Management Techniques

  • Beliefs about Grammar Teaching

• In Vivo Codes: Using the participants’ exact words.

  • "It just clicks" (Describing sudden understanding)

  • "Afraid to speak"

  • "Information overload"

  • "Practice makes perfect" (Stated belief)

• Process Codes: Capture actions, sequences, or changes over time.

  • Negotiating Meaning

  • Scaffolding Understanding

  • Building Rapport

  • Shifting Task Focus

  • Overcoming Communication Breakdown

• Analytical/Interpretive Codes: Go beyond description to infer meaning, assumptions, or theoretical links. These often emerge later in the process.

  • Learner Autonomy Displayed (Interpreting a student’s independent action)

  • Facilitating Interlanguage Development (Linking an observation to SLA theory)

  • Power Dynamics in Group Work (Analyzing interaction patterns)

  • Resistance to Teaching Method (Interpreting student behavior/comments)

  • Teacher Identity Performance (Analyzing how a teacher presents themselves)

Tools for Coding:

• Manual: Highlighters, colored pens, margins on printed transcripts, index cards, spreadsheets (like Excel or Google Sheets). Low-tech but effective, especially for smaller datasets. You can create columns for data snippets, codes, memos, etc.

• CAQDAS (Computer-Assisted Qualitative Data Analysis Software): Tools like NVivo, MAXQDA, ATLAS.ti, Dedoose.

  • Pros: Excellent for managing large datasets, easily applying/removing codes, searching and retrieving coded segments, visualizing relationships, writing memos linked to data/codes.

  • Cons: Can have a steep learning curve, can be expensive, they do not analyze the data for you! Your brain still does the interpretive work. They are organizational and retrieval tools.

Rigor in Coding:

How do you know your coding is trustworthy?

• Clear Codebook: Essential for consistency.

• Inter-Coder Reliability/Agreement: Have another researcher code a portion of the data using your codebook and compare results. Discuss discrepancies to refine definitions. (Even discussing with a colleague can be insightful).

• Memos: Write analytical memos throughout the process – notes to yourself about why you created a code, emerging thoughts, connections you see, problems you encounter. This documents your thinking process.

• Member Checking (Optional but valuable): Share your emerging themes or interpretations with some participants (if appropriate and feasible) to see if they resonate with their experiences.

• Peer Debriefing: Discuss your coding process and emerging themes with colleagues or supervisors.

Connecting Coding to Your Role as a Teacher Trainer:

Understanding coding helps you analyze data from classrooms or teacher development sessions more systematically. Imagine:

• Analyzing transcripts of mentoring sessions to identify common challenges novice teachers face (Classroom Management Anxiety, Lesson Planning Difficulties, Seeking Practical Strategies).

• Coding observation notes from experienced teachers’ classes to identify effective techniques for Error Correction, Concept Checking, or Managing Group Work.

• Analyzing teachers’ written reflections after a workshop to gauge their understanding and application of new methods (Applying TBLT Principles, Struggles with Implementation, Positive Learner Reactions).

The themes emerging from your coded data can directly inform the content and focus of your training sessions, making them more relevant and evidence-based. Furthermore, being able to articulate how you arrived at your conclusions (through systematic coding) adds credibility when you share your findings with the wider community of educators you engage with.

Coding is a skill that improves with practice. It requires patience, attention to detail, and a willingness to engage deeply and reflexively with your data. It’s the bridge between collecting rich descriptions of classroom life and developing meaningful insights that can advance our understanding of language teaching and learning. Good luck!

How will I analyze frequencies and duration?

All qualitative can be converted to quantitative data and vice versa. When conducting qualitative data, you might find it useful to convert data to quantitative data and then analyze it.

Converting qualitative data into quantitative data – sometimes called “quantitizing” – involves systematically assigning numerical values to non-numerical information like text or observations. This allows you to perform statistical analyses, summarize trends numerically, and create charts/graphs.

It’s crucial to remember that this process simplifies the richness of qualitative data and context can be lost. However, it can be very useful for identifying broad patterns, comparing groups, or tracking changes over time. The key is to have clear, predefined rules for the conversion.

Here are three detailed examples relevant to contexts like ELT teacher training:

Example 1: Quantifying Themes in Open-Ended Survey Feedback

• Goal: To identify and compare the frequency of different types of positive feedback trainees provided about a recent methodology workshop.

• Qualitative Data: Written responses to the open-ended survey question: “What did you find most valuable or positive about this workshop?” (Imagine you have 50 responses).

• Conversion Process:

  1. Qualitative Thematic Analysis: Read through all 50 responses. Identify recurring positive themes. Let’s say you identify themes like: Practical Teaching Ideas, Collaborative Activities, Trainer's Expertise, Useful Resources Provided, Clear Explanations.

  2. Develop Coding Rules: Define each theme clearly so you can consistently identify it.

  3. Quantitize using Presence/Absence: Create a spreadsheet. List each respondent (1-50) in rows. Create columns for each theme (Practical Ideas, Collaboration, Expertise, Resources, Clarity). Read each response again. If a response mentions elements related to ‘Practical Teaching Ideas’, put a ‘1’ in that column for that respondent. If it doesn’t, put a ‘0’. Do this for all themes and all respondents.

• Resulting Quantitative Data (Conceptual Table):

  Respondent	Practical ideas	Collaboration	Expertise	Resources
  Student 1	1	0	1
  Student 2	1	1	1

  Total Count: (e.g., 35) (e.g., 28) (e.g., 22) (e.g., 15) Percentages: (e.g., 70%) (e.g., 56%) (e.g., 44%) (e.g., 30%)

• Quantitative Analysis & Visualization:

  1. Frequency Table: Create a table showing the final counts and percentages for each theme (as shown conceptually above). This summarizes which positive aspects were mentioned most often.

  2. Bar Chart: Create a bar chart where each bar represents a theme, and the height of the bar indicates the frequency or percentage of respondents who mentioned it. This provides a clear visual comparison of theme prominence. Title: “Most Valued Aspects of Methodology Workshop (N=50)”.

  3. (Optional) Cross-Tabulation: If you have demographic data (e.g., years of teaching experience: 0-2 yrs vs. 3+ yrs), you could create a table comparing the percentage of each group that mentioned specific themes (e.g., “Percentage Mentioning ‘Practical Ideas’: Novice vs. Experienced Trainers”). This could reveal differences in priorities between groups.

Example 2: Quantifying Observed Teaching Practices

• Goal: To track the frequency of specific student engagement techniques used by trainee teachers during observed lessons over time.

• Qualitative Data: Structured observation notes or checklists completed by supervisors during classroom observations (e.g., 3 observations per trainee for 15 trainees). The notes describe techniques used.

• Conversion Process:

  1. Define Observable Behaviors: Specify the exact techniques you’re looking for (e.g., Used Pair Work, Used Group Work, Called on students by name, Used open-ended questions, Gave specific positive praise).

  2. Develop Rating Scale/Count: Decide how to quantify. You could:

     • Frequency Count: Count the number of times each specific behavior occurred per lesson (more complex).

     • Presence/Absence per Lesson: Simply note if the behavior occurred at least once during the lesson (Yes=1, No=0). (Let’s use this for simplicity).

  3. Quantitize: Review the observation notes/checklists for each lesson. In a spreadsheet, list each observation (Trainee 1-Obs 1, T1-Obs 2, T1-Obs 3, T2-Obs 1, etc.) in rows. Create columns for each behavior (Pair Work, Group Work, Named Call-on, Open Qs, Specific Praise). Enter ‘1’ if the behavior was observed in that lesson, ‘0’ if not.

Resulting Quantitative Data (Conceptual Table):

Observation ID	Pair work	Group work	Named call-on	Praise
Student ID	0	1	1	0
Student ID	1	1	0	0

• Quantitative Analysis & Visualization:

  1. Percentage Table: Calculate the overall percentage of all observed lessons where each behavior occurred.

  2. Bar Chart: Create a bar chart showing the overall percentage for each engagement technique across all observations. Title: “Frequency of Engagement Techniques Observed Across All Lessons”.

  3. Line Graph (Tracking Change): Calculate the percentage of lessons exhibiting each behavior separately for Observation 1, Observation 2, and Observation 3. Create a line graph with Observation # (1, 2, 3) on the X-axis and Percentage on the Y-axis. Plot separate lines for each behavior (e.g., one line for ‘Pair Work’, one for ‘Open Qs’). This visualizes trends in the use of techniques over the course of the training/practicum.

Example 3: Quantifying Sentiment in Interview Data

• Goal: To assess trainees’ overall sentiment (positive, negative, neutral) towards a newly introduced peer coaching program.

• Qualitative Data: Transcripts from semi-structured interviews where trainees discussed their experiences with the peer coaching program (e.g., 20 interviews).

• Conversion Process:

  1. Define Sentiment Categories & Rules: Create clear definitions for assigning an overall sentiment score to each trainee’s discussion about the program:

     • +1 (Positive): Expresses clear benefits, enjoyment, value; minimal reservations.

     • 0 (Neutral/Mixed): Expresses both significant positives and negatives; balanced view; non-committal.

     • 1 (Negative): Focuses on problems, frustrations, lack of value; significant criticism.

  2. Develop Coding Rules: Read through transcripts, highlighting segments related to peer coaching.

  3. Quantitize: Read all relevant segments for each trainee. Based on the overall tone and balance of comments according to your predefined rules, assign a single sentiment score (+1, 0, or -1) to each trainee.

• Resulting Quantitative Data (Conceptual Table):

  Name	Trainee ID	Sentiment score
  Sue	343433	1
  Rob	834333	-1
  Steve	831122	0

• Quantitative Analysis & Visualization:

  1. Frequency Table: Create a table showing the number and percentage of trainees assigned to each sentiment category (+1, 0, -1).

  2. Pie Chart: Create a pie chart visually representing the proportion of trainees in each sentiment category. Title: “Trainee Sentiment Towards Peer Coaching Program (N=20)”.

  3. Bar Chart: A simple bar chart showing the number/percentage for each category (Positive, Neutral, Negative) can also be very clear.

  4. (Optional) Link back to Qualitative: You could create a table showing common qualitative themes mentioned by trainees within each sentiment group (e.g., “Themes mentioned by trainees with POSITIVE sentiment: Learning from partner (8/12), Increased confidence (7/12)… Themes mentioned by trainees with NEGATIVE sentiment: Scheduling conflicts (3/3), Partner mismatch (2/3)…”). This combines the quantitative summary with qualitative detail.

These examples illustrate how qualitative data can be systematically converted into numbers, allowing for quantitative summaries and visualizations that can highlight key patterns and trends.

How might I analyze a questionnaire with Likert-scale items?

A Likert scale is a common type of psychometric scale used in questionnaires to measure attitudes, opinions, perceptions, or frequencies of behavior. It typically presents respondents with a statement (an “item”) and asks them to rate their level of agreement, frequency, importance, likelihood, etc., on a symmetrical, ordered scale.

• Structure: Consists of multiple items, all designed to measure different facets of the same underlying construct (e.g., motivation, anxiety, attitude).

• Response Options: Usually 5 or 7 points with labels that represent an ordered continuum (e.g., Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree; or Never, Rarely, Sometimes, Often, Always).

• Key Assumption (and Debate): While the categories are clearly ordered (ordinal), researchers often assume the psychological distance between each point is roughly equal, allowing them to treat the data as interval level for certain analyses, especially when combining multiple items into a composite score.

How to Analyze Likert Scale Data

Analysis depends on whether you’re looking at:

A. Individual Likert-type items (analyzing responses to single questions).

B. A composite Likert Scale score (combining scores from multiple related items).

A. Analyzing Individual Likert-Type Items (Treating as Ordinal Data)

This approach focuses on the distribution of responses for each specific question.

1. Descriptive Statistics:

   • Frequencies and Percentages: This is the most important first step. Calculate how many respondents chose each option (e.g., 10% Strongly Disagree, 20% Disagree, 15% Neutral, 40% Agree, 15% Strongly Agree). This shows the exact distribution of opinions on that specific item. Use Frequency Tables.

   • Mode: The most frequent response category. Identifies the most common sentiment or rating.

   • Median: The middle value when responses are ordered. It’s a good measure of central tendency for ordinal data, less affected by skewed distributions than the mean.

   • Range / Interquartile Range (IQR): Describe the spread or variability of responses.

2. Visualization:

   • Bar Charts: The best way to visualize the distribution of responses for a single item. Each bar represents a response category (e.g., Strongly Disagree to Strongly Agree), and the height shows the frequency or percentage.

   • Stacked Bar Charts: Useful for comparing response distributions across different items OR between different groups (e.g., comparing boys’ vs. girls’ responses to the same item). Can show percentages adding up to 100%.

   • Diverging Stacked Bar Charts: Can effectively visualize agreement vs. disagreement around a central neutral point.

B. Analyzing a Composite Likert Scale Score (Often Treated as Interval Data)

This involves combining responses from multiple related items designed to measure a single underlying construct (e.g., overall “Classroom Engagement”).

1. Data Preparation:

   • Assign Numerical Values: Consistently assign numbers to responses (e.g., 1=Strongly Disagree, 5=Strongly Agree).

   • Reverse Coding: If some items are negatively worded (e.g., “I find group work boring”), reverse their scores (1=5, 2=4, 3=3, 4=2, 5=1) before combining, so higher scores consistently mean higher levels of the construct.

   • Check Reliability: Crucially, before summing or averaging items, check if they reliably measure the same construct using statistics like Cronbach’s Alpha (aim for > 0.70 or 0.80). If reliability is low, combining scores is questionable.

   • Calculate Composite Score: Sum or average the (potentially reverse-coded) numerical scores across all relevant items for each respondent.

2. Descriptive Statistics (for the Composite Score):

   • Mean: Calculate the average composite score. Often interpreted as the average level of the construct (e.g., average classroom engagement score). Use with caution, acknowledging the interval assumption.

   • Standard Deviation: Measure the spread or variability of the composite scores around the mean.

   • Median, Range, IQR: Still valuable descriptors of the composite score’s distribution.

3. Visualization (for the Composite Score):

   • Histograms: Show the frequency distribution of the composite scores across the group.

   • Box Plots: Visualize the median, quartiles, range, and potential outliers of the composite score. Excellent for comparing the distribution between different groups (e.g., pre-test vs. post-test scores).

4. Inferential Statistics (Comparing Groups or Exploring Relationships):

   • ~~T-tests (Independent Samples): Compare the mean composite scores between two independent groups (e.g., compare mean engagement scores for students who participated in Project A vs. Project B).~~

   • ~~Paired Samples T-test: Compare the mean composite scores for the same group at two different time points (e.g., pre-intervention vs. post-intervention engagement scores).~~

   • ~~ANOVA (Analysis of Variance): Compare the mean composite scores between three or more groups (e.g., compare engagement scores across different classrooms or teaching methods).~~

   • ~~Correlation (e.g., Pearson): Examine the linear relationship between the composite Likert scale score and another continuous variable (e.g., is there a correlation between engagement score and final grade?).~~

Important Cautions:

• Ordinal vs. Interval: Strictly speaking, individual Likert items are ordinal. Calculating means on single items is statistically debated (though common). Composite scale scores are more likely to approximate interval properties, making means and related inferential statistics more justifiable, but it remains an assumption. Always report frequencies/percentages for individual items.

• Meaning of “Neutral”: Consider if “Neutral” represents true indifference, ambivalence, or an unwillingness to answer.

• Validity: Ensure your Likert scale instrument truly measures the concept you intend it to measure.

---

Classroom Research Examples:

Example 1: Assessing Student Perceptions of Classroom Climate

• Context: A teacher wants to understand students’ feelings about the learning environment in their Grade 8 English class at the end of a term.

• Instrument: A 10-item questionnaire using a 5-point Likert scale (1=Strongly Disagree, 5=Strongly Agree). Items include: “I feel comfortable asking questions in this class,” “The teacher respects students’ ideas,” “Students in this class are kind to each other,” “I find the lessons engaging.”

• Analysis Approach (Focus on Item-Level Descriptive Analysis):

  1. Frequencies & Percentages: Calculate the percentage of students selecting each response (1-5) for every single item. Put this in a Frequency Table.

  2. Visualization: Create Bar Charts for key items (e.g., “I feel comfortable asking questions,” “Students are kind”). This visually shows the distribution of agreement/disagreement. Maybe use a diverging stacked bar chart centered on “Neutral”.

  3. Central Tendency: Calculate the Median and Mode for each item. Does the mode indicate general agreement or disagreement? Is the median at 3 (Neutral), 4 (Agree), etc.?

  4. Reporting: “Regarding classroom climate, analysis focused on response distributions for each item. For instance, 78% of students ‘Agreed’ or ‘Strongly Agreed’ with the statement ‘The teacher respects students’ ideas’ (Median=4, Mode=4). However, responses were more varied for ‘Students in this class are kind to each other’, with 25% selecting ‘Disagree’ or ‘Strongly Disagree’ (Median=3, Mode=4).” Accompany with key bar charts.

Example 2: Measuring Change in Student Motivation for Reading Before and After an Intervention

• Context: A teacher implements a new 6-week independent reading program with choice elements and wants to see if it impacts students’ motivation for reading.

• Instrument: A validated 12-item Reading Motivation Scale using a 4-point Likert scale (1=Almost Never, 4=Almost Always). Items group into sub-constructs like Intrinsic Motivation (e.g., “I read because I enjoy it”) and Extrinsic Motivation (e.g., “I read because my teacher assigns it”). The teacher administers it the week before the program (Pre) and the week after (Post).

• Analysis Approach (Focus on Composite Score & Pre/Post Comparison):

  1. Reliability Check: Calculate Cronbach’s Alpha for the Intrinsic Motivation items and separately for the Extrinsic Motivation items (using Pre-test data, for example) to ensure reliability.

  2. Composite Scores: Calculate an average Intrinsic Motivation score and an average Extrinsic Motivation score for each student at both Pre-test and Post-test time points (after reverse-coding any necessary items).

  3. Descriptive Statistics: Calculate the Mean and Standard Deviation for the average Intrinsic and Extrinsic scores at Pre-test and Post-test.

  4. Inferential Statistics: Conduct Paired Samples T-tests: one comparing the mean Intrinsic Motivation score (Pre vs. Post) and another comparing the mean Extrinsic Motivation score (Pre vs. Post). This tests if the changes observed are statistically significant.

  5. Visualization: Create Box Plots showing the distribution of Intrinsic Motivation scores side-by-side for Pre-test and Post-test. Do the same for Extrinsic Motivation.

  6. Reporting: “A paired-samples t-test revealed a statistically significant increase in students’ mean intrinsic reading motivation scores from Pre-test (M = 2.85, SD = 0.65) to Post-test (M = 3.25, SD = 0.55), t(df)=…, p < .001. No significant change was found for extrinsic motivation (p = .250). Box plots illustrating these distributions are shown in Figure 1.” Interpret the results in light of the reading program.

These examples show how you can analyze Likert data descriptively at the item level or combine items into scales for more complex analyses, including tracking changes or comparing groups within a classroom setting.

How might I do a correlative study?

Correlational research is a type of non-experimental research design where the researcher measures two or more variables and assesses the statistical relationship (i.e., the correlation) between them with little or no effort to control extraneous variables.

• Purpose: To determine if, and to what extent, variables are associated or co-vary. Does a change in one variable correspond to a change in another? In what direction (positive or negative)? How strong is the association?

• Key Output: Often involves calculating a correlation coefficient, such as Pearson’s r (for interval/ratio data) or Spearman’s rho (for ordinal data), which ranges from -1.0 (perfect negative correlation) through 0 (no correlation) to +1.0 (perfect positive correlation).

• Crucial Limitation: Correlation does NOT imply causation! This is the most important takeaway. Just because two variables are related (e.g., ice cream sales and crime rates are positively correlated) doesn’t mean one causes the other. There might be a third variable (e.g., hot weather) influencing both. Correlational studies identify relationships, they don’t explain why they exist in terms of cause-and-effect.

Analyzing Qualitative Data in a Correlational Framework:

Directly calculating statistical correlations with purely qualitative data (like interview transcripts or detailed observation notes) isn’t standard because the data isn’t numerical. However, you can explore relationships in ways inspired by correlational thinking:

1. Quantitizing Qualitative Data:

   • Process: Convert aspects of your qualitative data into numbers. As discussed previously, this involves coding your data (e.g., interview transcripts, observation notes) for the presence/absence (0/1) or frequency of specific themes, behaviors, sentiments, etc.

   • Analysis: You can then calculate statistical correlations between these newly created numerical variables and other quantitative variables (like test scores, attendance rates, or Likert scale scores).

   • Example: You code observation notes for the frequency of teacher “wait time” instances after questions. You could then correlate this frequency count with students’ scores on a subsequent quiz.

   • Caveat: This simplifies the qualitative data significantly.

2. Qualitative Exploration of Co-occurrence/Patterns:

   • Process: Instead of statistics, you systematically look for patterns within your qualitative data. Do certain themes, experiences, or characteristics frequently appear together in your interviews or observations?

   • Analysis: Use techniques like thematic analysis or matrix displays. For example, create a matrix where rows are participants and columns are key themes. Examining the matrix helps see which themes tend to cluster for particular participants or groups.

   • Example: Analyzing interview transcripts, you might notice that trainees who express high levels of “teaching anxiety” also consistently describe challenges related to “classroom management” and “lack of confidence with technology.” You report this observed association using descriptive narrative and supporting quotes.

   • Reporting: You describe the relationship qualitatively, highlighting the pattern of co-occurrence, rather than reporting a numerical coefficient.

Analyzing Likert Scale Data in a Correlational Framework:

This is more straightforward as Likert scale data is numerical (or often treated as such).

1. Using Composite Scores (Treated as Interval):

   • Process: Calculate composite scores (sum or average) for different Likert scales measuring distinct constructs (e.g., ‘Student Motivation Scale’, ‘Student Self-Efficacy Scale’). Ensure scales are reliable (e.g., Cronbach’s Alpha).

   • Analysis: Use Pearson’s correlation coefficient (r) to assess the linear relationship between the two composite scores. A scatter plot is excellent for visualizing this relationship.

   • Example: Correlate students’ scores on a ‘Classroom Engagement’ Likert scale with their scores on an ‘Attitude Towards School’ Likert scale.

2. Using Individual Items (Ordinal):

   • Process: If you want to correlate responses on two single Likert-type items (treating them as ordinal data).

   • Analysis: Use Spearman’s rank-order correlation coefficient (rho).

Examples in a Qualitative Classroom Study Context:

Often, a study might be primarily qualitative (using interviews, observations) but incorporate Likert scales or quantitized data to explore specific relationships as part of a broader investigation.

Example 1: Exploring the Connection Between Teacher Enthusiasm (Observed) and Student Engagement (Likert Scale & Observation)

• Overall Study Goal: To qualitatively understand how variations in observable teacher enthusiasm impact student engagement in a high school history class.

• Qualitative Data: Detailed classroom observation field notes describing teacher behaviors coded for enthusiasm (e.g., vocal variety, movement, expressive gestures, positive statements) and student behaviors coded for engagement (e.g., asking questions, participating in discussions, on-task behavior, body language). Interviews with students about what makes them feel engaged.

• Correlational Element:

  1. Quantitize Observation: Develop a rating for overall observed ‘Teacher Enthusiasm’ (e.g., Low=1, Medium=2, High=3) for different lesson segments based on coding criteria applied to field notes. Similarly, rate overall observed ‘Student Engagement’ (Low=1, Med=2, High=3) for those segments.

  2. Likert Scale Data: Administer a short Likert scale after specific lessons asking students about their perceived engagement (‘I felt engaged during this lesson’, ‘I found this lesson interesting’ - Strongly Disagree to Strongly Agree). Calculate an average ‘Self-Reported Engagement’ score per student per lesson observed.

  3. Analysis:

     • Calculate Spearman’s correlation between the ordinal ‘Observed Teacher Enthusiasm’ rating and the ordinal ‘Observed Student Engagement’ rating across lesson segments.

     • Calculate Pearson’s correlation between the ‘Observed Teacher Enthusiasm’ rating (could treat as pseudo-interval or use Spearman’s) and the average ‘Self-Reported Engagement’ Likert score for the corresponding lesson.

  • Qualitative Integration: Use the rich observation notes and interview data to explain the correlations. What specific enthusiastic behaviors seemed linked to which specific engagement behaviors? What did students say about how teacher enthusiasm affected their engagement? The qualitative data gives meaning to the numerical relationship.

Example 2: Investigating the Relationship Between Student Self-Perception as Readers (Interview) and Reading Strategy Use (Observation/Documents)

• Overall Study Goal: To qualitatively explore how middle school students perceive themselves as readers and how they use reading strategies during independent reading.

• Qualitative Data: Semi-structured interviews with students about their reading habits, preferences, and how they see themselves as readers (e.g., “good reader,” “struggling reader,” “improving reader”). Observations during reading time, noting strategy use (e.g., predicting, summarizing, questioning, using graphic organizers). Analysis of student reading logs or response journals (documents) for evidence of strategy use.

• Correlational Element:

  1. Quantitize Interview Data: Based on the interview analysis, categorize each student’s overall ‘Reading Self-Perception’ (e.g., Positive=3, Mixed/Neutral=2, Negative=1). This is quantitizing a key qualitative theme.

  2. Quantitize Strategy Use: Based on observations and journal analysis, create a score for ‘Observed Reading Strategy Use Frequency/Variety’ (e.g., count the number of different strategies observed/noted per student over a period, or rate frequency on a simple scale: Low=1, Medium=2, High=3).

  3. Analysis: Calculate Spearman’s correlation between the ordinal ‘Reading Self-Perception’ category and the ‘Strategy Use’ score/category.

  • Qualitative Integration: Dive deep into the interview and observation data. How do students with positive self-perceptions describe their reading process and what specific strategies are they observed using effectively? Conversely, what challenges and strategy use patterns are seen among those with negative self-perceptions? The qualitative data explains how self-perception and strategy use might be linked.

In these examples, the core study remains qualitative, aiming for deep understanding. The correlational analysis serves as one tool within that broader study to identify or confirm patterns of association, which are then explored and explained using the rich qualitative data.

How might I use GenAI to help analyze my qualitative data?

https://odysee.com/@benjaminstewart:d/GenAI-demo-analysis_10042025:0

Demonstration

GenAI Prompt: Act as a qualitative research assistant. I will provide student reflections that relate to their weekly progress for conducting their research project. Please read them carefully and identify and organize into three levels of coding by grouping initial codes into categories and then a separate list that groups categories into themes:

GenAI Output based on prompt above

Based on the student reflections provided, here is a qualitative analysis organized into three levels of coding: Initial Codes, Categories, and Themes.

---

Level 1: Initial Codes

(Direct actions, specific challenges, plans, or feelings mentioned by students)

• Made changes to literature review

• Corrected comments on literature review

• Designed instruments (questionnaire, interview, observation sheet)

• Planning to correct instruments

• Planning to apply instruments

• Anticipating challenge: good instrument design

• Anticipating challenge: writing analysis/results

• Planning to use tutor comments

• Checking participant group size/suitability

• Completed observations

• Discussed strategies with participants

• Pending interviews

• Planning interviews

• Complication: festivities impacting observation

• Have relevant info, want more

• Close to finishing observations

• Checking amount of information needed

• Continued observation

• Haven’t modified lit review (time constraint)

• Tried classifying/identifying information

• Plan: finish literature review

• No observation next week (school events)

• Challenge: no classes for students

• Focusing on lit review changes

• Plan: start data analysis

• Observed limited hours (quiz/teacher injury)

• Administered questionnaire (adapted)

• Interviewed teacher (second time)

• Struggling: getting lesson plans

• Plan: message teachers for lesson plans

• Haven’t observed formal feedback

• Teacher plans feedback (without instrument)

• Finished data collection (interviews, lesson plans)

• Corrected literature review

• Unavailable next week

• Needs pointers for remote work (data analysis)

• Challenge: scheduling/doing enough work remotely

• Wants talk on data analysis

• Listened to podcast (helpful, clarified ideas)

• Observed classes

• Brief interview (longer one rescheduled due to emergency)

• Got school info (syllabus, timetable)

• Plan: continue/longer observations

• Plan: longer interview

• Plan: review/revise literature review

• Plan: start data analysis

• Plan: triangulate data/lit review

• Challenge: potential schedule conflict (graduation pics)

• Worry: finishing on time

• Worry: writing analysis correctly

• Making plans/taking time

• Completed data collection (observations of specific strategies)

• Observed strategies (images, audio, differentiated work)

• Plan: get more info (teaching system, study plans)

• Plan: complete research questions

• Challenge: observation difficulties (activities)

• Challenge: getting desired interviews

• Plan: observe as much as possible

• Plan: conduct appropriate interviews

• Seeking tutor help (grammar, research)

• Observed classes (Linguatec)

• Took audio recordings

• Wrote detailed observation notes

• Worry: number of classes/student flow

• Plan: observe more classes

• Administrator keeping student updated

• Challenge: spring break closing impacting observation

• Got lesson plans

• Got teacher background info

• Observed lessons (essay feedback)

• Observed peer feedback process

• Got syllabus

• Got pictures of final essays

• Plan: observe teacher feedback lesson

• Plan: interview teacher (digital vs. written feedback)

• Plan: interview students (peer feedback rationale, reflection)

• Challenge: selecting students for interview

• Idea: compare high/low score students for interviews

• Seeking tutor advice (interview selection)

• Plan: analyze data (categorize by RQ)

• Continued observations/interviews

• Understood teacher approaches/strategies

• Plan: finish interviews

• Plan: start writing methodology section

• Challenge: missing final project stages (school break)

• Plan: ask teachers directly in interviews (plans, beliefs, outcomes)

• Refining interview questions

• Plan: modify/finish literature review (align with data)

• Observed more classes (different activities)

• Collected information

• Plan: finish observations

• Plan: collect all information

• Plan: start writing next part

• Challenge: strategies might not relate to lit review

• Plan: review lit review examples

• Plan: make changes to lit review if needed

• Plan: connect examples/observations

• Plan: finish examples

• Plan: connect all parts

• Changed keypoint/RQ (due to feasibility issues)

• Adjusted keypoint (teacher questions during feedback)

• Problem: bad internet/lack of storage

• Plan: correct literature review (reflecting change)

• Plan: enhance instrument information

• Plan: find more instruments (teacher/students)

• Plan: triangulate information

• Plan: organize document (messy)

• Challenge: intimidated by literature review

• Worry: finding best way to triangulate

• Strategy: work one step at a time

• Strategy: focus on weekly necessary tasks

• Appreciates help

• Checked phone storage

• Modified lit review (coherence/organization)

• Visited institute/started observations

• Plan: visit second school

• Plan: talk to authorities (permission to record/interview)

• Challenge: second school might deny access

• Challenge: fixing literature review

• Worry: technology problems (storage)

• Plan: delete apps/check storage

• Plan: visit more schools (contingency)

• Observed classes (high school/extension)

• Collected data (worksheets, presentations, materials)

• Plan: continue observing/notes/recording

• Plan: apply questionnaire (more responses needed)

• Plan: change literature review

• Plan: change RQs

• Plan: start writing methodology

• Challenge: deciding RQ changes

• Seeking help/suggestions

---

Level 2: Categories

(Grouping related initial codes into broader categories)

1. Data Collection Activities:

   • Observing classes (various settings)

   • Conducting interviews (teachers, students)

   • Administering questionnaires/instruments

   • Collecting documents (lesson plans, syllabus, essays, materials, school info)

   • Recording sessions (audio)

   • Taking detailed notes

   • Visiting research sites/schools

   • Seeking permissions/access

2. Literature Review Work:

   • Making changes/corrections based on comments/data

   • Reviewing/revising/modifying

   • Aligning lit review with data/observations

   • Improving coherence/organization

   • Feeling intimidated by the task

   • Connecting examples to observations

3. Data Analysis & Interpretation:

   • Starting data analysis

   • Classifying/categorizing information

   • Planning for analysis (remote work, by RQ)

   • Triangulating data (planning, worrying about method)

   • Understanding teacher approaches/strategies

   • Connecting observations to lit review/examples

4. Research Writing & Documentation:

   • Writing methodology section

   • Writing analysis/results section

   • Organizing research document

   • Documenting observations (notes, recordings)

   • Completing research questions

5. Planning & Scheduling:

   • Weekly planning (observations, interviews, writing)

   • Setting goals for next 7 days and beyond

   • Scheduling tasks around constraints (events, breaks, personal schedule)

   • Developing contingency plans (visiting more schools)

   • Focusing on specific tasks sequentially (“one step at a time”)

   • Taking time to do things right

6. Challenges & Obstacles:

   • Logistical issues (scheduling conflicts, school events, breaks, teacher availability, participant numbers, site access denial)

   • Data collection problems (instrument design, getting lesson plans, observing specific events like feedback, low questionnaire response)

   • Technical difficulties (internet connection, phone storage)

   • Analysis/Writing challenges (how to write analysis, relating data to lit review, triangulation method, deciding RQ changes)

   • Time constraints & workload management (lack of time for lit review, balancing tasks, working remotely)

   • Emotional/Motivational factors (feeling intimidated by lit review, worry about finishing on time, worry about quality)

7. Adaptation & Problem-Solving:

   • Changing RQs/keypoints based on feasibility

   • Adjusting plans due to unforeseen circumstances (rescheduling interviews, focusing on lit review when observation isn’t possible)

   • Refining instruments/interview questions

   • Developing strategies to overcome obstacles (messaging teachers, asking directly in interviews, comparing student groups, checking storage, having backup schools)

   • Checking sufficiency/relevance of collected data

8. Seeking & Utilizing Support:

   • Using tutor comments/advice

   • Seeking help/suggestions/pointers (from tutor, for data analysis)

   • Appreciating help received

   • Using external resources (podcast)

   • Relying on administrator updates

9. Progress & Achievement:

   • Completing tasks (observations, interviews, data collection)

   • Making progress on writing/revisions

   • Feeling close to finishing stages (e.g., observation)

   • Achieving understanding (e.g., teacher approaches)

   • Successfully gathering needed information/documents

---

Level 3: Themes

(Grouping related categories into overarching themes)

1. Executing the Research Process: This theme encompasses the core activities involved in conducting the research project.

   • Categories included: Data Collection Activities, Literature Review Work, Data Analysis & Interpretation, Research Writing & Documentation.

2. Managing Project Logistics and Contingencies: This theme covers the practical aspects of planning, scheduling, anticipating problems, and adjusting the research plan in response to real-world constraints and challenges.

   • Categories included: Planning & Scheduling, Challenges & Obstacles, Adaptation & Problem-Solving.

3. Researcher Development and Resourcefulness: This theme focuses on the student researchers’ personal experiences, their sense of progress, their reliance on external support systems, and their reflections on the process.

   • Categories included: Seeking & Utilizing Support, Progress & Achievement, (aspects of Challenges & Obstacles related to personal feelings/worries).

---

This three-level coding structure moves from specific reported actions and thoughts (Level 1) to grouped activities and issues (Level 2), finally culminating in broader themes that describe the major dimensions of the students’ research experience as reflected in their weekly updates (Level 3).

Crucial Preliminary Notes for Novice Researchers:

1. LLMs are Assistants, NOT Researchers: These tools can process text rapidly, identify patterns, summarize, and even generate code, but they lack true understanding, context, and critical thinking. You, the researcher, are always in control. You must guide the LLM, critically evaluate its output, and make the final decisions based on your knowledge and research goals.

2. Data Privacy and Ethics are Paramount: NEVER input identifiable student data (names, specific locations, highly unique personal stories) into public LLMs. Anonymize your data thoroughly before pasting any text. Check your institution’s policies regarding the use of AI tools with research data. You are responsible for protecting participant confidentiality.

3. Potential for Bias and “Hallucinations”: LLMs can sometimes generate plausible-sounding but incorrect information (hallucinations) or reflect biases present in their training data. Always cross-reference, validate, and critically assess the output.

4. Start Small and Verify: Use LLMs for specific, manageable tasks first. Always double-check calculations and interpretations.

5. Focus on Augmentation, Not Replacement: Think of the LLM as a tool to speed up certain processes (like initial coding ideas, summarizing text, calculating basic frequencies) or to help you learn (explaining concepts), not as a replacement for rigorous qualitative methodology or statistical software for complex analyses.

---

Tutorial: Using LLMs as a Research Assistant

Part 1: Assistance with Coding Qualitative Data (e.g., Interview Excerpts, Open-Ended Survey Responses)

• Goal: To get initial ideas for codes or themes present in your textual data.

• How the LLM Helps: Quickly processes text to suggest potential patterns you might miss or to apply a preliminary coding structure.

• Process & Example:

  1. Prepare Your Data: Select a small batch of anonymized text excerpts. For example, 5-10 student answers to: “What did you find most difficult about the group research project?”

  2. Craft Your Prompt (Inductive - Finding Themes):

     • "Act as a qualitative research assistant. I will provide [number] anonymized student responses about challenges in a group research project. Please read them carefully and identify the main themes or categories of challenges mentioned. For each theme, provide a brief descriptive label and list the specific responses (or key phrases) that fit under it."

     • [Paste your anonymized responses here]

  3. Craft Your Prompt (Deductive - Applying Codes):

     • "Act as a qualitative research assistant. I have a preliminary coding scheme for challenges in group work: [Code A: Communication Problems], [Code B: Uneven Workload], [Code C: Scheduling Conflicts]. Please read the following [number] anonymized student responses. Identify specific sentences or phrases that align with each code and list them under the corresponding code label."

     • [Paste your anonymized responses here]

  4. Analyze LLM Output: The LLM will provide a list of potential themes or coded segments.

  5. Your Critical Role (MOST IMPORTANT):

     • Review: Read the LLM’s suggestions carefully alongside your original data.

     • Refine: Do the themes make sense? Are they distinct? Do they accurately represent the data? Rename themes for clarity. Merge similar themes. Split broad themes. Create new themes the LLM missed.

     • Verify Coding: Does the text listed under each code truly fit? Move miscategorized text. Ensure consistency in how codes are applied across all responses.

     • Finalize: You create the final, validated codebook and ensure all data is coded reliably according to your refined scheme. The LLM only provided a draft.

Part 2: Assistance with Generating Descriptive Statistics from Qualitative Data

• Goal: To calculate frequencies or percentages of codes/themes once you have finalized the coding.

• How the LLM Helps: Performs simple calculations based on structured input.

• Process & Example:

  1. Prepare Your Data (Structured Format): After coding (with or without LLM help) and validating, you need to quantify. Create a summary table or list. For example, if you coded 50 responses and found ‘Theme X’ appeared in 20 responses, ‘Theme Y’ in 35, and ‘Theme Z’ in 10:

     • Option A (List): Theme X: 20, Theme Y: 35, Theme Z: 10. Total Responses: 50.

     • Option B (Table - simplified):
       | Theme | Frequency |
       | :------ | :-------- |
       | Theme X | 20 |
       | Theme Y | 35 |
       | Theme Z | 10 |
       (Total N=50)

  2. Craft Your Prompt:

     • "Act as a data analysis assistant. I have coded 50 qualitative responses and found the following theme frequencies: Theme X appeared 20 times, Theme Y appeared 35 times, and Theme Z appeared 10 times. Please calculate the percentage for each theme based on the total number of responses (N=50)."

     • (Or, reference your table if you can paste/describe it clearly)

  3. Analyze LLM Output: The LLM will provide the percentages (e.g., Theme X: 40%, Theme Y: 70%, Theme Z: 20%).

  4. Your Critical Role:

     • Verify: Double-check the simple calculations.

     • Contextualize: Remember these percentages represent the frequency of mentioning a theme, not necessarily its importance or intensity.

     • Visualize: Use these numbers to create your own charts (e.g., a Bar Chart). You can ask the LLM: "What's the best type of chart to show the frequencies of these themes?" but create the chart yourself using spreadsheet software or other tools.

Part 3: Assistance with Analyzing Likert Scale Questionnaires

• Goal: To calculate descriptive statistics for Likert items or scales and get suggestions for visualization or further analysis.

• How the LLM Helps: Performs calculations (frequencies, percentages, maybe basic means if appropriate), explains concepts, suggests chart types or statistical tests.

• Process & Example:

  1. Prepare Your Data: Tally your responses. Do not paste raw data from individual students. Have summaries ready.

     • Item Example: For an item “I enjoyed the reading activity” (1=SD, 5=SA), your summary is: 1=2 students, 2=5 students, 3=8 students, 4=15 students, 5=10 students (Total N=40).

     • Scale Example: You calculated a composite ‘Reading Enjoyment’ score for 40 students; the mean score is 3.8, standard deviation is 0.9.

  2. Craft Your Prompt (Item Analysis):

     • "Act as a research assistant analyzing Likert data. For an item scored 1(Strongly Disagree) to 5(Strongly Agree), the frequencies for N=40 students were: Score 1: 2 students; Score 2: 5 students; Score 3: 8 students; Score 4: 15 students; Score 5: 10 students. Calculate the percentage for each score category. What is the mode and median response? What type of chart best visualizes this?"

  3. Craft Your Prompt (Scale Analysis):

     • "Act as a research assistant. I have composite scores from a 'Reading Enjoyment' Likert scale (N=40 students). The mean score is 3.8 (SD=0.9). How would I typically interpret this mean score (scale is 1-5)? What does the standard deviation tell me? What chart could show the distribution of these composite scores?"

  4. Analyze LLM Output: The LLM will provide percentages, identify mode/median, interpret the mean/SD in general terms, and suggest charts (e.g., bar chart for item, histogram for scale).

  5. Your Critical Role:

     • Verify Calculations.

     • Evaluate Suggestions: Does the interpretation make sense for your specific scale? Is the suggested chart appropriate?

     • Perform Further Analysis: If the LLM suggests a test (e.g., “To compare pre/post scores, you could use a paired t-test”), you need to understand that test and perform it using statistical software (Excel, SPSS, R, Python). You could ask the LLM to explain the t-test or even generate code for R/Python, but you must validate the code and run it yourself.

Part 4: Assistance with Correlative Analysis

• Goal: To explore potential relationships between variables (e.g., Likert scores, quantitized codes, performance data) and understand correlation concepts.

• How the LLM Helps: Explains correlation, interprets coefficients, suggests visualization, potentially generates analysis code.

• Process & Example:

  1. Prepare Your Data: Have pairs of data ready for comparison (e.g., a list of students’ ‘Motivation Scale’ scores and their corresponding ‘Project Grades’; frequency counts of ‘Observed Strategy Use’ paired with ‘Reading Test Scores’). Summarize or anonymize fully.

  2. Craft Your Prompt (Understanding/Interpretation):

     • "Act as a research assistant. Explain the concept of Pearson's correlation (r). If I found a correlation of r = +0.65 between a 'Student Engagement' Likert scale score and 'Time Spent on Homework' (measured in hours), how would I interpret this relationship in terms of direction and strength? What can I NOT conclude from this correlation?"

     • "Act as a research assistant. A statistical analysis showed a Spearman correlation of rho = -0.30, p = 0.04 between 'Level of Math Anxiety' (Likert scale) and 'Frequency of Asking Questions' (observed count). Explain what this result means regarding the relationship, its strength, and statistical significance."

  3. Craft Your Prompt (Code Generation - Use with Caution):

     • "Act as a data analysis assistant using Python. I have two lists: student_engagement_scores = [list of scores] and student_grades = [corresponding list of grades]. Generate Python code using the 'scipy.stats' library to calculate the Pearson correlation coefficient and p-value between these lists."

  4. Analyze LLM Output: The LLM provides explanations, interpretations, or code.

  5. Your Critical Role:

     • Understand Concepts: Ensure you grasp the explanations of correlation, significance, and especially the correlation vs. causation limitation.

     • Validate Code: If you use generated code, meticulously check it for correctness before running it with your data in a proper environment (Python, R).

     • Interpret Holistically: Combine statistical findings with your qualitative understanding. Why might that correlation exist in your specific classroom context? What do your interviews or observations add?

---

Conclusion: The Researcher Drives the Research

LLMs can be powerful assistants for novice researchers, helping to manage text, generate initial ideas, perform basic calculations, and explain concepts. However, they are tools, not oracles. Your critical judgment, methodological understanding, ethical responsibility, and deep engagement with your data are irreplaceable. Always validate, always question, and always place the LLM’s output within the broader context of your research goals and ethical obligations.