How powerful could this framework become if we applied it to assessment of an observation of a student engaging in a rich learning task. I know many of you who use action stations and rich learning tasks in your classrooms much of this will sound familiar.
This is my attempt at synthesising CIT with educational teaching practice to consider a different approach to assessment of and for learning.
The critical incident technique (CIT) is a well-established qualitative research tool described by John C. Flanagan in 1954. It is a flexible set of principles that can be modified and adapted to meet specific requirements. By gathering factual reports made by observers, we can build a picture of the students thinking and capabilities that we are investigating. The CIT format effectively turning anecdotes into data.
To gather really useful, meaningful information, about how and what students are thinking and feel about their learning, anecdotal information regarding these areas is plentiful throughout action stations. However, the anecdote’s subjective nature makes it difficult to access and credibly analyse using traditional quantitative research methods.
We need to carefully consider this framework to ensure we are consistent in what we are looking for and how we will navigate conversations to gain this information.
These steps are essential components of CIT;
Step 1 PURPOSE - Identify what it is that you want to know as the students educator.
Students actively engaging in a rich math task are working and applying what they know or seeking ways to solve their problems. Sometimes the decisions and actions performed during learning in context results in "critical incidents" which may be either a success or a failure. This interview technique is used to help identify the specific actions (behaviours), decisions, and information which led to the critical incident.
Possible purpose of the interview. "We want to learn more about how you make decisions as you work out this rich math task."
They may also be derived from;
- problem solving strategies applied
- number knowledge applied to the learning process
- strategies applied
- key competencies
- learner dispositions
- areas of strength or interest
- ability to communicate the learning process
- ability to build and construct knowledge with / alongside others
Step 2 Collecting data - Through probing questioning we can help the student to talk about identifying the critical incidences they experienced throughout the task. These are decisions that may have or would likely have resulted in the success or an error in completing the task. It is important that the focus remains on the incident and what led to it.
These should be recorded as close as possible to the time when they occurred. During and throughout the activity.
Memory of the learning will become improved if they know they have to report / share to an audience.
During the questioning;
- Ask for clarification, justification, explanation of their thinking and actions.
- Avoid discussion about things not related to the learning process.
In Action Stations we record anecdotal notes on a class list for simplicity and ease of access.
Step 3 Analyzing the data
Often considered the most important and difficult step. With a framework such as this we are able to then summarise and describe the data so it can be used for practical purposes.
The aim is to increase the usefulness of the data without sacrificing comprehensiveness, specificity, or detail- Know the purpose of assessment
- Used a consistent classification system - national standards, levels, stages, strategies, progression framework, KC's, learner dispositions.
- Developing a set of success criteria. This can emerge from the observations made and work towards developing next step. In action stations this is developed through KC's every Monday but as an educator and the student as a learner we need to know more than just that - the ability to identify the learning in a task.
- Placing the observations into the above categories will require experience and judgement from the teacher.
Step 5 - Interpreting and reporting
For each decision point before adding to classification register, consider the following:
- Errors If an error occurred, what was it?
- Optimal How should the decision have been made?
- Ambiguous What information could have helped make the decision. Was any information missing?
- Error Avoidance Could the error have been avoided? If so, how?
- Environmental Factors What aspects of your environment influenced your decision?
- Expert / Novice Do (or would) experts and novices differ in their decision making?
- Information What information was used in making the decision? How was it obtained?
- Ongoing Training What would you teach them about this kind of incident in the next lessons?
Advantages
- Information is gathered directly from the students
- Can follow-up on statements in future lessons
- Can interview multiple students for a more complete perspective
- One of the most significant advantages of the CIT is its connection to real-world problems and situations provided through the words of the participant, thus limiting the subjectivity of the researcher (Kain, 2004).
- allows participants a wide a range of responses within one rich task
- participants freely develop the context using their own perspective, allowing cultural neutrality (Gremler, 2004)
- When recalling incidents, participants openly use their language This can also become a disadvantage if ESOL or difficulty with oral language or self confidence.
- operates with a flexible set of rules to let themes or theories emerge directly from the data - no preconceived ideas about challenges student may or may not have.
Disadvantages
- Subject to the interpretation of teachers / students
- Questioning needs to be conducted shortly after or during a task
- Memory about an incident may be biased or fallible - will require Teacher competency in questioning, recording and analysing of narratives.
- Some students may be reluctant to talk about certain elements of the process. Requires a relationship of trust, and feel safe to take risks.
Linking theory to practice is a great way to help define and refine elements that make it work, both being as vital as each other.
Journal of Dental Education March 1, 2008 vol. 72 no. 3 299-304
Sourced from: http://www.jdentaled.org/content/72/3/299.full
http://www.hr-survey.com/Critical_Incident_Interview_Guide.htm
http://www.jorgvianden.com/uploads/2/2/7/7/22771362/vianden_2012_cit.pdf
