Learning Technologies Review: e-Portfolio and e-PDP Evaluation and Recommendations

Executive Summary

Part 1 of the Learning Technologies review deferred a decision on providing a single institutional e-Portfolio/e-PDP platform. Provision was made for continuation of trials, started in 2008, of the Pebblepad e-Portfolio toolkit. A 3000 user hosted service licence was purchased from Pebble Learning. This report makes recommendations to the Learning Technologies review based on evaluation of the Pebblepad pilot project and drawing on other initiatives in the institution and from evidence of practice across the HE sector.

The Pebblepad pilot project has delivered a robust implementation of Pebblepad, embedded into the institutional VLE with single sign on access authenticated via LDAP. The Pebblepad toolkit supports a range of curriculum developments in relation to both e-Portfolio and e-PDP. Project leaders are generally of the view that curriculum design is more influential that the usability of the software in fostering student engagement. Students engage well, where there is clear purpose and relevance to professional skills development and employability.

Experience from the Pebblepad pilot projects, other institutional initiatives and research across the HE sector would strongly suggest that one size does not fit all in terms e-Portfolio and e-PDP provision. An independent evaluation concluded that Pebblepad and Mahara represent the most balanced e-Portfolio products currently available (Himpsl and Baumgartner, 2008).

General recommendations

1. It is essential that a decision is made that clarifies the mid to long term institutional provision of a platform to support e-Portfolio and e-PDP. This will enable programme teams to make plans in relation to the 5 year review cycle that can be implemented with confidence in the stability of the provision.

2. Provision should be embedded within the institutional VLE/myMMU portal including single sign in.

3. Technical and administrative support for the institutional e-Portfolio/e-PDP should be fully integrated within the overall institutional VLE support infrastructure.

4. Staff training should adopt a similar mechanism to staff training and engagement for Moodle i.e. based on a locally facilitated cascade model.

5. Student training should be designed into local level IT and skills training as determined by programme teams.

Specific recommendations

6. Mahara should be adopted as the core e-Portfolio/e-PDP provided and supported by the institution.

7. Programmes that can make a good case for, and effectively support, the use of Pebblepad, or other systems already in use, for professional development and critical reflective practice should be supported by the institution in doing so.

8. A series of mid to long term evaluations should be undertaken to monitor and compare the effectiveness of e-Portfolio and e-PDP systems in use at MMU.

9. Horizon scanning pilot projects should be undertaken as new technologies emerge to support e-Portfolio and e-PDP with a view to expanding the range of tools available.

10. Institutional e-Portfolio and e-PDP provision should place an emphasis on student ownership of content, whist at MMU and beyond, and further work should be initiated in relation to MMU’s support of alumni.

Dr W. Rod Cullen Principal Lecturer in Learning and Teaching Technologies Pebblepad Pilot Project Leader Centre for Learning and Teaching

Scope of this report

The purpose of this report is to make recommendations, to the Learning Technology Review, for the provision of medium to long term support for e-Portfolio and e-PDP at MMU. The report will focus on the evaluation of the Pebblepad pilot project (ongoing since June 2008) but will draw, where possible, on other initiatives in the institution and from evidence of practice across the HE sector.

Introduction and background

Whilst the two concepts of student portfolios and personal development planning (PDP) may be linked, and the terms often used interchangeably, they can also be considered as disparate. For example, it is entirely possible to have a PDP process that does not utilise portfolios, or to use academic portfolios for purposes other than PDP (Brett et al, 2009).

Portfolios can take many different forms. In Higher Education portfolios are generally a collection of student work representing evidence of learning, achievement and development as part of a unit, course or programme of study, although there is considerable variation in the way these are put into practice in different subject areas (Arter & Spandel, 1992, Baume 2001a, 2001b, Strivens, 2006). The QAA (2008) defines PDP as “a structured and supported process undertaken by an individual to reflect upon their own learning, performance and/or achievement and to plan for their personal, educational and career development”. Central to the idea of PDP are the processes of self-audit, action planning, reflection and review.

Traditionally, student portfolios have been paper based, often resulting in large, cumbersome text based documents. Increasingly students work in all subject areas is completed and presented in a wide range of digital formats that go beyond text and may include images, audio and video formats as the norm. The need to include such elements as part of portfolios has lead to the development of software tools that facilitate the aggregation and sharing of multi-media content - the e-Portfolio. e-Portfolio tools allow students to record and collate evidence of their academic work and development (learning outcomes, skills or competencies) upon which they can reflect, use to plan future work and easily share in a range of formats. As such they provide a potentially powerful technological toolkit to support personal development planning - the e-PDP.

With national requirements for structured and systematic support for PDP by UK university students (QAA 2000), it is not surprising that notions of e-PDP and e-Portfolio have emerged to describe institutions’ attempts to provide suitable technological support. Recent studies provide strong evidence that the success (or otherwise) of e-PDP or e-Portfolio approaches depends on well defined concepts of PDP and portfolio that are appreciated by staff and students, and embedded within the curriculum (Strivens, 2007). Technology that supports notions of both e-Portfolio and e-PDP is desirable, but when scoping solutions it is essential to remember that e-Portfolio and e-PDP are simply tools; the focus must be on designing learning opportunities that enable students to construct useful portfolios and which facilitates the PDP processes.

e-Portfolio and e-PDP at MMU

In the past, MMU has investigated several software products for e-PDP and e-Portfolios. ePet, developed at the University of Newcastle, and the e-Portfolio system developed by WebCT (now Blackboard), the providers of our current virtual learning environment, have both proved unsuitable after small scale evaluations across a number of programmes. Elgg was briefly considered during 2007 but was not pursued due to the significant technical requirements for initial implementation. In 2007, MMU began a pilot of the Pebblepad e-Portfolio toolkit. Initially with a 1000 user license (now increased to 3000 users) a series of 9 pilot projects were set up to evaluate this technology. This work is ongoing. The Windows Livetoolkit (student email and cloud facilities) was included in the evaluations of the first part of the Learning Technologies Review, however, this has proved problematic as currently there is no interface available between the Windows Live toolkit and the staff MS Outlook Exchange email system. Staff could be provided with separate Windows Live accounts but this would result in them having two MMU email infrastructures that could prove confusing and difficult to manage. Integration of the staff and student system was briefly discussed part of the Outlook Exchange project but this was not considered in scope of the project at that time.

The first part of the Learning Technologies Review deferred the decision on institutional support for e-Portfolio and e-PDP pending further investigation and evaluation. In the absence of a clear institutional policy or technology platform, or guidance about the mid to long term future of PebblePad, various Departmental or Faculty based initiatives have explored options in terms of e-Portfolio or e-PDP software. For example, MMUBS has made some use of TalentOnView software through the SRC project, to enable students to showcase their skills, although discussion with staff involved suggest that results have been disappointing (Robin Johnston SLTF MMUBS, personal communication).

Yet another factor to be considered is that coming from employers in terms of industries looking for creative skills such as art and design industries and those involved in creative digital industries such as broadcasting and web advertising. From work done on the JISC SRC project, the ProDev days bringing students and employers together, and considerable networking of MMU staff with local digital industries, employers do not welcome the concept of an institutional e-portfolio. Rather they want to see graduates showcase their own creative portfolio, employers seeing an institutional e-portfolio as something which is constraining students and making it more difficult to identify graduates who have the most appropriate creative talents.

Pebblepad evaluation

The following evaluation is a summary based on the following activities:

• 9 Preliminary pilot project reports (March 2009)

• Project leader preliminary report (April 2009)

• 6 End of year 1 project reports (August 2009)

• Pilot project evaluation focus group (November 2009)

• e-PDP and e-Portfolio requirements document submitted to Learning Technologies Review Pt1 (December 2009)

• 7 One-to-one telephone interviews with pilot project leaders (July/August 2010)

• Publications and conference papers based on the pilot projects e.g. Overfield et al., (2009), Overfield and Ainley, (2009), Gough and Hamshire, (2010a, 2010b), Hamshire (2009, 2010, 2011a, 2011b)

Technical implementation and operation of Pebblepad

MMU is currently utilising a 3000 user, hosted service at www.pebblepad.co.uk/mmu. A “Power Link” enables users logged into WebCT to access their Pebblepad accounts directly without logging in again. In the early stages of the project this Power Link did not function reliably, however, this problem has been resolved over the last twelve months. Direct access to Pebblepad is also available through the myMMU portal.

Pebblepad accounts are created automatically on first login, which is authenticated through LDAP. In principle, anyone with a valid MMU user account can login and automatically set up a Pebblepad account. However, in order to limit access to 3000 licensed users, an additional “access group” is maintained within the Pebblepad admin system. Currently this places a high and unscalable administrative burden on the project leader.

Pebblepad “Gateways” offer a formal mechanism that can be used by users to share assets that they have created for purposes of assessment, grading, provision of feedback or archiving of student work. Several projects have made use of this functionality and it has proved to be very robust if a little complex to set up and manage. At the moment there is no direct link from Gateways to the student record system but this could be developed assuming appropriate technical and administrative responsibilities are allocated.

Where Pebblepad has been used for formal assessment, MMU user accounts have been created for external examiners, (e.g. the PG Cert in Academic Practice) enabling remote access to students’ work in Pebblepad. This reduces the need to mail out copies of large portfolio folders in advance of exam boards and has been very well received by externals.

The Pebblepad service has proved to be robust and reliable and virtually no down time has been reported by users and project leaders. A small number of staff and student users have reported usability problems relating to the Flash interface and the click heavy, web based functionality of Pebblepad, as well as slow operational speeds both on and off campus. Overall, this has not been a major problem and technical support from Pebblepad has been exemplary.

Staff engagement

Pebblepad provided project teams with a wide range of structured, student centred tools that have been used to facilitate reflective practice, planning, critical thinking, evidence gathering and sharing. On the whole, project leaders are of the opinion that Pebblepad represents much more than an e-Portfolio and provides a great number of options in relation to reflective practice and e-PDP.

Several projects report difficulties in relation to buy in from all colleagues within a teaching team. The issue centred on technical abilities and training, communication of purpose, curriculum design, programme leadership and institutional commitment to Pebblepad (or other appropriate e-Portfolio /e-PDP platform). Discussion at the November 2009 project leader’s focus group agreed that leadership from management, communication of purpose to the teaching team, appropriate technical training and clear responsibilities for admin and technical support are key success factors for staff buy in.

There is a general feeling among the project leaders that staff buy in to Pebblepad, as a tool, is closely linked to staff perceptions of portfolio and/or PDP per se. Where there is a well-defined portfolio or PDP process, designed and embedded into the curriculum, Pebblepad can easily be embedded to support and enhance it, but this is much more difficult where portfolio/PDP is ill defined in the curriculum as a whole.

Several project leaders expressed a view that training for staff was essential. However, as Pebbelpad offers such a broad range of tools for a broad range of purposes, training should be targeted in relation to the specific set of tools being deployed to support the curriculum. This is analogous to the training plans for Moodle where it has been decided that beyond the provision of generic resources, the best place for training to be designed and delivered is at the level of the subject or programme team.

The Pebblepad user licence has been limited to 3000 users since September 2009 and there is currently no capacity for expanding projects. For valid reasons, the first stage of the Learning Technologies Review, in February 2010, deferred making a decision on an institutional e-Portfolio /e-PDP platform until early 2011. These combined factors, have made it very difficult for project teams to plan for the mid to long term (most project leaders refer to planning over the 5 year review cycle). This has now become the key barrier to further development of the pilot projects and e-Portfolio and e-PDP initiatives in general as colleagues are reluctant to invest time and effort to future development without mid to long term stability in the e-Portfolio /e-PDP platform.

Student engagement

All of the projects indicated that students, as well as staff, required some level of training in the use of Pebblepad. Some students reported that Pebblepad was confusing to navigate and that the web-based Flash interface could be click heavy for some tasks. In addition, as Pebblepad provides such a broad range of tools it is very important that students are clear which of the tools of the tools available they will be required to use in their studies. Training should be targeted at those tools which are being utilised within that programme, rather than providing generic Pebbelpad training. Overall students’ perceptions of Pebblepad have been positive.

Login and access problems, although rare, have proved difficult to resolve, especially where this has resulted from LDAP account problems. Most project leaders highlighted that it is important to be able to resolve such issues rapidly as a bad first impression for students often leads to a poor perception of the software and poor engagement.

There is some evidence to suggest that those students who are less engaged with their courses in general, are less positive about Pebblepad and vice-versa. Two project leaders were of the opinion that Pebblepad appears to be of most use to those students who were the most motivated, ie. those that could be expected to succeed no matter what.

Curriculum design was reported as having a greater impact on student engagement than aspects of the usability of the Pebblepad software. Use of Pebblepad was most significant where students had a very clear understanding of why they were using Pebblepad from the outset; what they were expected to produce with Pebblepad; and where the use of Pebblepad was embedded as a requirement rather than an optional bolt on activity within the curriculum. Some students who were initially sceptical about the use of Pebblepad became more engaged following specific sessions on employability. These sessions focused on how Pebblepad could be used to create CVs and share evidence of specific skills with prospective employers.

Specific Case Study

The pilot project that represents the most detailed evaluation of the use of Pebblepad at MMU is in the Department of Physiotherapy. This programme is a strand of the JISC funded SRC curriculum design project that is piloting agile, demand-led, curriculum design processes that promote flexible delivery and enhance student employability and is linked closely to the EQAL initiative. This project encapsulates many of the issues faced by the institution in selecting a platform for e-Portfolio and e-PDP and as such is worthy of specific attention here. In 2008, Pebblepad was adopted, as a mechanism for recording continuous professional development by the professional body for Physiotherapy (the Chartered Society of Physiotherapy (CSP), (CSP, 2008). The provision of Pebblepad is therefore considered an essential professional obligation by the Physiotherapy programme team. A key skills framework has been developed for the programme which is aligned with the standards of the Health Professions Council (HPC) and the Level 5 Career framework for Health and the CSP. The framework was designed to use the Pebblepad toolkit as the mechanism by which students evidence their academic work, skills, reflections and personal development plans. An important technical aspect is the ability in Pebblepad to “tag” evidence in relation to specific skills identified within the framework. Initial student evaluations of this project are very positive (Gough and Hamshire, (2010a, 2010b), Hamshire (2009, 2010)). This highlights the importance of designing e-PDP that is perceived by students as highly relevant to them in terms of employability, professional skills development and the role that a tool like Pebblepad can play in enabling this.

All of the projects emphasized that student ownership of the contents of e-Portfolios was essential to foster engagement. MMU Pebblepad accounts can be migrated to a personal account with Pebblepad after graduation and can be kept for up to 2 years without charge and after this time for a fee of £12 per year. In addition, Pebblepad is Leap2A (JISC, 2010) compliant allowing students to migrate the contents of their Pebblepad portfolios to other compliant systems (including ePet, Mahara and MyProgressFile) should they wish to do so.

Some project leaders also pointed out that their students attached a high level of importance to the “institutional identity” of their e-Portfolios as this attaches a level of kudos to their potential engagement with employers.

A particular strength of Pebblepad was the range of group management and collaborative tools that enable dialogue between tutors and students and between peers (one-to-one, one-to-many and many-to-one). These tools were especially useful for the provision of rapid formative feedback.

Moodle and e-Portfolio and e-PDP

One of the main drivers for the institution to move to the Moodle VLE is that it can easily integrate with complementary technologies and as such supports the ethos of EQAL. It is recognized that there are a range of e-Portfolio and e-PDP options which the institution could explore in this respect. It is not the purpose of this report to review all of the software options available, however, a brief comparison of Pebblepad with one significant possible option, Mahara will emphasise some of the key issues.

Pebblepad has recently released a new Moodle “block” enabling Moodle users to export items directly into their Pebblepad account (Pebblepad, 2011). Users can sign into Moodle and move to their Pebblepad account without the need to sign in again. Moodle blogs, forums, assignments and resources can be exported to the user’s Pebblepad account and Pebblepad assets including thoughts, abilities, activities and files can be created within Moodle and sent directly to Pebblepad. This helps to make managing files and activities more seamless.

As part of MMU’s Moodle implementation, some preliminary investigation is taking place into the Mahara e-Portfolio system, which is reported to be highly compatible and easily integrated within Moodle (Mahara, 2011). Mahara provides a single-sign on capability that allows users, at the option of the administrator, to be automatically logged in to both their Mahara and Moodle accounts by providing a username and password at only one of these sites (Mahara, 2011). The user can sign on at Mahara, and click on a link to their Moodle account, or sign on at Moodle, and click on a link to their Mahara account. At this stage, MMU does not have a working implementation of Mahara to evaluate fully this software.

Table 1 - Mahara and Pebblepad feature comparison

Pebblepad provides a wider range of tools, especially in relation to structured reflection than Mahara (see Table 1). In addition, several Pebblepad tools (e.g. Forms, Profiles and Webfolios) are highly customizable by tutors giving further design flexibility. It is not, as yet, been possible to assess the degree to which tools can be customized in Mahara.

In a detailed evaluation of 60 e-Portfolio s providers, Himpsl and Baumgartner (2008) concluded that Pebblepad and Mahara represent the most balanced products, which can be used for portfolio work without huge time expenditure for installation. Both systems require some acclimatization effort but, once their logic is clear, they are easy to handle. Since this paper was published both products have continued to develop and evolve and they are still the front runners in the market.

Both Pebblepad and Mahara can be downloaded, installed and run as a local institutional service with the obvious cost implication in relation to hardware and technical/administrative support. Both can also be provided as a hosted service (Pebblepad via Pebble Learning and Mahara via various organisations, including our Moodle hosts ULCC.) However, Pebblepad is commercially licensed whereas Mahara is Open Source software.

Key feedback to the Learning Technologies Review

The following feedback in an update to that provided to part 1 of the LTR in 2010

• Whilst systems such as Pebblepad and Mahara have undoubted strengths, all e-Portfolio/e-PDP systems have limitations when faced with the challenge of enabling university-provided structure and feedback on student-owned data. Experience from the Pebblepad pilot projects, other institutional initiatives and research across the HE sector would suggest very strongly that one size does not fit all in terms of e-Portfolio and e-PDP provision.

• e-Portfolio and e-PDP have distinct but overlapping requirements.

• If students are to invest in these systems, there must be value at university (for assessment) and beyond (for job and professional body membership applications), and the students must be able to own, access and control e-Portfolio/ePDP data at university and after graduation.

• Ideally, university staff should offer structure to (in the form of suggested tag lists and PDP scaffolding, such as reflective prompts) and feedback on, student-owned data that can be tagged to facilitate its retrieval when required.

• e-Portfolio tools need to support the collaborative production of content in a range of media formats by groups of learners.

• e-PDP tools need to support scaffolded reflection and feedback (formative and summative) from invited tutors /peers / mentors / coaches

• In the current financial environment, it is clear that a hosted Pebblepad service is the least cost effective option to the institution as a whole. Mahara does appear to offer a viable, more cost effective alternative tool set to many of the projects that are already engaged in e-Portfolio and e-PDP, however, this would not be without significant redevelopment work. It should be noted that Pebblepad does provide some tools (which are currently being used) for which there is no equivalent in Mahara. Furthermore at least one programme at MMU considers the provision of Pebblepad to be an essential professional obligation.

Learning Technologies Review Criteria

LTR part 1 set out the following rationale and review criteria.

The technologies we choose must deliver the right things (fitness to vision) at the right time (fitness to transformation timeline) in the right way (robust and scalable) at the right price (value-for-money) whilst preserving our capacity for change (avoiding lock-in). We therefore propose these as review criteria:

• Fitness to vision

• Fitness to transformation timeline

• Robustness and scalability

• Value-for-money

• Avoiding lock-in

Table 3 is an update, based on the investigations undertaken, of the evaluation made as part of LTR part 1. Three options are evaluated for institutional level provision. It should be noted that the evaluation of Mahara is tentative and based on secondary sources.

Table 2 - e-Portfolio and e-PDP updated evaluation table

Note:Scores on a scale 1-5 of increasing fit

Recommendations

General recommendations

1. It is essential that a decision is made that clarifies the mid to long term institutional provision of a platform to support e-Portfolio and e-PDP. This will enable programme teams to make plans in relation to the 5 year review cycle that can be implemented with confidence in the stability of the provision.

2. Provision should be embedded within the institutional VLE/myMMU portal including single sign in.

3. Technical and administrative support for the institutional e-Portfolio/e-PDP should be fully integrated within the overall institutional VLE support infrastructure.

4. Staff training should adopt a similar mechanism to staff training and engagement for Moodle i.e. based on a locally facilitated cascade model

5. Student training should be designed into local level IT and skills training as determined by programme teams.

Specific recommendations

6. Mahara should be adopted as the core e-Portfolio/e-PDP provided and supported by the institution.

7. Programmes that can make a good case for, and effectively support, the use of