11. Evaluation Frameworks & Subjectivity


Evaluation Frameworks & Subjectivity

In our last session, Rachel, Abdullah and I gave a presentation on how we had gone about creating a framework for evaluating materials, and then applied this framework to a coursebook (English Unlimited B1).

I’ve included a gallery below of our powerpoint slides, but first a few thoughts that arose during our and others’ presentations.

  • The importance of aesthetic

While creating our framework, we had some rather lengthy discussions on the importance of the aesthetic of a coursebook.  When I talk about the aesthetic of materials, I don’t just mean illustrations and photographs, but also the layout and formatting of the text, as well as diagrams, colours, etc.  Eventually we agreed that the visuals and overall aesthetic should engage and motivate learners, and spent some time during the evaluation process considering different types of visuals.

When another group presented their framework on Thursday, they had included the question ‘are the pictures visually attractive?’  While I agree that this is important, for me, it’s not enough.  I believe we need to go further in considering the didactic, affective and retentive quality of visuals in materials.

  • The need for dialogue

It was really interesting to hear about, see and compare the frameworks that other groups had come up with and their evaluations of the same coursebook and unit.  Some of the differences in our evaluations highlighted the need for dialogue between teachers, the implication being that evaluation processes should be done collaboratively.

  • Subjectivity

This brings me to the issue of subjectitvity.  While Sheldon describes coursebook evaluation as subjective (Sheldon 1988:245), Roberts counters that ‘subjectivity is arbitrary and has no rational defence’ (Roberts 1996:383), instead promoting judgement as being more beneficial.  While I agree that judgement (an opinion that you have after thinking carefully about something) plays an integral role in evaluating materials, I don’t think we can be so quick to dismiss subjectivity.  Here’s why:

If subjectivity is about  a person’s feelings and thoughts that no one else can know directly or completely, it provides only one perspective. That perspective is inherently related to the context in which that viewer is working as well as their own experiences and understanding of ELT.  The individual context is perhaps the only position from which they can truly judge, and arguably – if evaluating materials for use in their own classroom – the only position from which they should.

But this point relates directly back to the issue above of collaborative evaluation.  One person’s perspective shows only a small part of the picture.  But combined with the perspective of others’, creates a bigger picture.  Objectivity, then, is borne of multiple subjectivities.

  • Universality

Because of the very different contexts in which Rachel, Abdullah and I are working, we tried to create a framework which could be applied in these different contexts.  As such we created our own evaluative criteria based on our own principles, rather than adapting frameworks from the reading (although we considered aspects of these at various stages throughout).  Because we’d tried to create something more universal, our framework seemed quite open to interpretation (and more so in comparison to the frameworks others had created).  When creating it we stressed the need for teachers to define clearly what they understood from each of the criteria.  This is an issue of personalisation – for a framework to be truly constructive, it needs to be specific to the context of the individuals using it.  A universal framework then, is only useful as a launch pad for discussion, a trigger for dialogue between teachers working in the same or similar contexts.

  • Sheldon, L. E. (1988) Evaluating ELT textbooks and materials. ELT Journal 42 (4): pp.237-246
  • Roberts, J. T. (1996) Demystifying materials evaluation. System 24 (3): pp. 375-389. doi: 10.1016/0346- 251X(96)00029-2.


Our presentation slides, with some explanatory notes for the reader:

Materials Evaluation-page-001Because Rachel, Abdullah and I all teach in very different contexts, we wanted to create a universal framework, that would be accessible and usable for a variety of teaching contexts.  However, it was important to us that the framework somehow recognised the importance of local context.Materials Evaluation-page-002

Materials Evaluation-page-003So to begin with we felt we should identify some common ground for what we felt should be considered in evaluating materials, and use these common ideas to create a framework whose criteria could then be applied with a specific teaching context in mind.Materials Evaluation-page-004We looked at a range of existing frameworks as well as the principles for materials development that we’d come up with during an earlier input session (see Materials should be…).  Once we had a collection of potential criteria, we then categorised and prioritised these.  We wanted the framework to be both qualitative and quantitative, and so also decided at this point that we should include a scale as well as a space for comments.Materials Evaluation-page-005Our approach was challenging, but fun, and quite revealing as it required us to consider very carefully both the semantics of certain terms as well as their relevance to an evaluation framework.  Working on this collaboratively was also beneficial, as it meant we had to work to articulate and justify our beliefs and feelings about what elements were important.Materials Evaluation-page-006Eventually we agreed on certain non-negotiable points that we felt should be included, and identified and discussed our personal top-ten criteria that we felt relevant to our individual teaching contexts.  The overlapping criteria then became the basis of our framework.Materials Evaluation-page-007Once we’d agreed on our criteria, we then expanded these into specific questions to use in a framework.Materials Evaluation-page-008We ended up with eleven questions…

… and then created a scale for teachers to use in answering the questions.  We deliberately wanted to use a six point scale here, to prevent teachers from just going with the middle ground, and thus encourage them to think more deeply about the questions.Materials Evaluation-page-009We then decided to order the questions according to what a teacher might notice or look at first in a coursebook.

Here’s what our final framework looked like:Materials Evaluation-page-010In close up:Materials Evaluation-page-011We included the question about adaptability to make the framework more appropriate for a variety of contexts.  A teacher might answer no to a specific question, but feel confident that although the book wasn’t providing a certain element, this was something they could adapt or develop accordingly.  This also made allowance for the varying degrees of experience teachers have.Materials Evaluation-page-012Materials Evaluation-page-013Materials Evaluation-page-014A point we felt particularly important to remember, was that even in answering questions in a framework, teachers views remain – at least to some extent – subjective, a point which I discussed in more detail above.Materials Evaluation-page-015Having created the framework, we then turned our attention to evaluating a coursebook (English Unlimited, B1+).

Our first step – as for most teachers, I believe, when choosing a book – was to have a look through and see if there was anything that interested or struck us about the book.  One particular element that jumped out was the inclusion of a ‘culture’ focus at the end of every second unit.  While we might have liked this to be more integrated, and ideally in every unit, we acknowledged that what had been included was of interest and value to learners.

Materials Evaluation-page-016

We also had a close look at the visuals used throughout the book.  Generally we felt the visuals were valuable, but in some cases felt the images to be a little patronising in that they were unrealistic (staged) or childish in quality.Materials Evaluation-page-017We then used the framework we’d created to evaluate both the coursebook as a whole…Materials Evaluation-page-018And a single unit (unit 4)…Materials Evaluation-page-019Materials Evaluation-page-020Materials Evaluation-page-021Materials Evaluation-page-022Materials Evaluation-page-023Interestingly, our average score for the coursebook as a whole was higher than our average score for unit 4.  What was particularly interesting in applying the framework to this coursebook, is that on first glance I probably wouldn’t have chosen this book, but having considered it in more detail, with the aid of a framework, I’d be more inclined to use it.Materials Evaluation-page-025


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s