09 Feb GENERATION evaluation
Design processes are often seen as being immeasurable. Maybe because ‘designers’ believe the impact is obvious – “Look how happy they all are?!” (etc!)
When it comes to using the design process in the context of ‘business’ then the bottom may well speak for itself. However, if design is going to be taken seriously in areas where the bottom line is neither financial nor a product on a shelf then it is up to the designer to find ways of demonstrating the value in the things we find most meaningful.
So how DO you measure the impact in a story?
25 YEARS OF CONTEMPORARY ART – Co-production Projects
I ran a workshop in Glasgow last week to demonstrate an evaluation process I had developed. GENERATION have run a series of co-production arts projects across Scotland for young people. We wanted to carry the co-production ethic through into the evaluation and capture meaningful reflections on the process – and capture them in a way that would demonstrate the impact the projects had had.
The process I designed is based on storyboarding methods usually used in idea generation. Instead I used the storyboards as a means of reflection – documenting an individual’s story or journey through the project. I’ll get into the process in a moment but firstly, here’s a film of how it went down…
When evaluating projects the challenge is gathering QUALITATIVE feedback that can be QUANTITATIVELY measured. To achieve this I devised a method of rating stories against their relevance to the impacts defined by GENERATION. The impacts are areas of development and learning that GENERATION set out to achieve through the various projects. The impacts are: Skills, Confidence, Relationships, and Positive Progression).
In addition to these impacts we also agreed to look out for other common themes that were identified as being important to the young people through their feedback.
Above: One of the BIG questions – five questions around the room that helped warm everyone up for the tasks ahead.
Below: Time lapse film of the Big Question session.
My evaluation process was to be delivered to the participants (young people) from each of the projects by the artists and educators (practitioners) who had worked with them throughout the project. But first the practitioners needed to be experience it for themselves – this was the purpose of my workshop. I would help GENERATION evaluate the practitioners experience by using my process and at the same time they would experience it in preparation for delivering it themselves back at their own projects.
It was important to me that the rating didn’t devalue experiences that didn’t appear to meet the impacts – it was important that the young people shared their story in their words, in their language. It was up to us to find what we were looking for.
For example: If a young person told us that they had made friends as a result of attending one of the programmes and that they enjoyed making things for this exhibition that they had never done before, then we tagged that story with relationships and skills.
During one of the project evaluations several young people told us that having somewhere [the studio where we met] away from school to be creative was important and this was identified as an additional impact.
So the young people were not made aware of the impacts before we gathered their stories because it was important that the stories were authentic and not influenced by the targets of the project.
We wanted their story, from their perspective and in their words.
ABOVE: TO IDENTIFY WHO-SAID-WHAT WHILE STILL HAVING THE FREEDOM TO HAVE FEEDBACK GATHERED TOGETHER WE GAVE EACH PARTNER PROJECT THEIR OWN COLOUR OF PEN. PARTNERS INCLUDED ‘THE NATIONAL GALLERY’, ‘MCMANUS’ AND THE ‘DUNDEE CONTEMPORARY ARTS’.
I devised a simple matrix to gather impact summaries of the stories. This was simply a grid with the person’s name and a box for each of the GENERATION impacts and another box where we recorded additional impacts and notes. We graded the tags using three sizes of dot. If, for example, a reference was made to relationships (either explicitly or in the participant’s own words) then we marked a dot. If the story made more than one mention of relationships we made a larger dot. If the whole story was about relationships then we filled the box with a dot. In this was, we could see at a glance, which tags were most significant.
As in any evaluation of this kind there was a level of subjectivity. For this reason it is recommended that the people who cary out the project evaluations are the people who have been companions on the learning journey with the young people – the youth workers and practitioners. They are best placed to interpret the stories (of a year or more) most accurately through their knowledge of both the project and the young person.
Above: Inside the CCA, Glasgow
At the end of the day the practitioners left the CCA with a tool kit under their arm and experience under their belt – along with the confidence (fingers crossed) to evaluate their own projects with their own young people.
I personally had the opportunity to use the method in Nov/Dec last year to evaluate the projects at The McManus and the Dundee Contemporary Arts and it went very well.
Through design, testing and training, the whole evaluation has been an extremely rewarding and valuable process for me and I’m looking forward to seeing results from the other five centres over the coming months.
Design processes are often seen as being immeasurable which results in qualitative feedback and stories being viewed as less valuable than other more obvious forms of measurement. It is therefore the responsibility of the designer to find ways of demonstrating the value in what we believe to be meaningful. In this way we will only add value to the process and more meaning to the impact.
I would love to hear from you if you have come across qualitative measurement that has impressed you.
Location: Glasgow Centre for Contemporary Arts
Film and editing: Jon Gill © 2015