OLT_LaTS summary_Criterion Wkshop - Western Sydney University


CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information        

Not  for  further  publication  in  present  form  

  A  national  project  funded  by  the  Office  for  Learning  and  Teaching,  DIISRTE     A  sector-­‐wide  model  for  assuring  final  year  subject  and  program  achievement  standards   through  inter-­‐university  moderation  (SP10-­‐1843)     Executive  Summary  and  Draft  Recommendations  for  discussion  and  feedback  

  Lead  institution:  University  of  Western  Sydney       Partner  institutions:  Australian  National  University,  Charles  Darwin  University,  Griffith  University,   LaTrobe  University,  Macquarie  University,  Queensland  University  of  Technology,  The  University  of   Melbourne     Project  leaders  and  contact  details:   Prof.  Kerri-­‐Lee  Krause  |  E:  [email protected]  |  M:  0434  189  407         Emeritus  Prof.  Geoff  Scott  |  E:  [email protected]     Team  members:  A/Prof  Heather  Alexander  (Griffith  University),  Prof  Tom  Angelo  (La  Trobe   University),  Emer.  Prof  Stuart  Campbell  (University  of  Western  Sydney),  A/Prof  Martin  Carroll   (Charles  Darwin  University),  Prof  Liz  Deane,  A/Prof  Duncan  Nulty  (Griffith  University),  Prof  Pip   Pattison  (The  University  of  Melbourne),  Prof  Belinda  Probert,  Prof  Judyth  Sachs  (Macquarie   University),  A/Prof  Ian  Solomonides  (Macquarie  University),  Prof  Suzi  Vaughan  (Queensland   University  of  Technology)     Status  of  this  consultation  paper:     • This  paper  has  been  prepared  for  discussion  at  various  fora  during  February/March  2013.  It   includes  the  Executive  Summary  and  Recommendations  that  will  appear  in  the  project’s  final   report,  due  for  submission  in  April  2013.  These  have  been  discussed  with  representatives  of   the  Office  for  Learning  and  Teaching  but  the  final  report  has  not  yet  been  formally   submitted.     • Also  appended  is  a  discussion  paper  (Deane  &  Krause,  2013)  proposing  a  possible  learning   standards  framework.  The  thinking  behind  this  paper  has  been  informed  by  the  outcomes  of   the  peer  review  of  standards  project  and  related  projects  in  the  sector.     • The  purpose  of  this  paper  is  to  provide  the  sector  with  an  update  on  project  outcomes,  to   foster  discussion  and  to  seek  feedback.     Compiled  by  Kerri-­‐Lee  Krause  and  Geoff  Scott   University  of  Western  Sydney   with  full  acknowledgement  and  deep  gratitude  to  our  project  team,  advisors  and  academic   colleagues  who  participated  as  peer  reviewers  and  provided  invaluable  feedback   25  February  2013         Funding  for  this  project  was  provided  by  the  Australian  Government  Office  for  Learning  and  Teaching.  The  views  expressed   in  this  document  do  not  necessarily  reflect  the  views  of  the  Australian  Government  Office  for  Learning  and  Teaching.  



CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information  

1 Executive  summary     This  report  documents  the  outcomes  of  the  project  entitled  A  sector-­‐wide  model  for  assuring  final   year  subject  and  program  achievement  standards  through  inter-­‐university  moderation,  supported  by   the  Australian  Government  Office  for  Learning  and  Teaching.  The  project  was  carried  out  over  two   years,  led  by  Professors  Kerri-­‐Lee  Krause  and  Geoff  Scott  (University  of  Western  Sydney),  together   with  11  project  team  members,  most  of  whom  are  senior  leaders  from  eight  Australian  universities1.   The  project  was  supported  by  a  national  steering  group  comprising  experts  and  key  stakeholders2,  an   international  expert  advisor,  Dr  Peter  Ewell  (National  Council  for  Higher  Education  Management   Systems,  USA).  Emeritus  Professor  Adrian  Lee  was  the  project  evaluator  and  critical  friend.         The  primary  aim  of  the  project  was  to  produce  a  valid,  reliable,  feasible  and  cost-­‐effective  method  of   assuring  the  quality  and  relevance  of  student  outcomes  and  their  assessment  across  parallel   discipline  areas  in  Australian  higher  education,  while  at  the  same  time  promoting  sector  diversity  and   responsiveness.  The  project  set  out  to  develop  and  validate  an  inter-­‐university  model  for  external   validation  of  assessment  inputs  and  processes,  together  with  validation  of  outcomes  in  the  form  of   assessment  grading  in  final  year  undergraduate  units/subjects  in  the  context  of  whole-­‐of-­‐degree   program  outcomes.  If  the  quality  of  Australian  higher  education  graduates  is  to  be  assured,  the   sector  needs  a  mechanism  to  assure  not  just  the  reliability  and  quality  of  marking;  it  also  needs   robust  evidence  that  what  is  being  given  focus  in  assessment  is  also  valid  and  reliable.       The  project  established  a  series  of  discipline-­‐based  peer  review  teams  to  review  anonymously  the   standards  of  the  learning  and  assessment  inputs  (identified  in  this  project  as  ‘teaching  standards’)   and  outcomes  (‘learning  standards’)  in  common  final  year  undergraduate  units  of  study  across  12   discipline  areas  in  eight  universities.  A  further  three  universities  joined  in  phase  2  of  the  project,  thus   reinforcing  the  potential  application  of  this  approach  beyond  the  parameters  of  the  original  project.   The  12  disciplines  involved  in  this  blind  peer  review  process  were:  Chinese,  Civil  Engineering,   Economics,  Environmental  Science,  History,  Journalism,  Law,  Marketing,  Music,  Nursing,  Philosophy,   and  Physics.     The  peer  review  process  involved  teams  of  three  academic  staff  –  one  from  the  home  university  and   two  partners  from  different  universities.  Teams  were  allocated  by  the  project  manager  with  every   effort  made  to  ensure  that  groups  represented  a  range  of  university  types.  The  home  university   provided  deidentified  course  information,  unit  guides,  assessment  information  and  unmarked   assessment  items  for  review.  The  two  peer  reviewers  undertook  a  guided  evaluation  of  the  quality   and  standards  of  the  deidentified  course  and  unit  inputs  and  graded  the  deidentified,  unmarked   assessment  artefacts  (outcomes)  in  each  selected  unit.  Peer  reviewers  did  not  know  the  name  of  the   university  from  which  the  unit  guide  and  assessment  items  had  come,  but  all  team  members  were   involved  in  teaching  and  assessing  an  equivalent  unit  in  their  home  institution.                                                                                                                               1

 Project  team  members:  A/Prof  Heather  Alexander  (Griffith  University),  Prof  Tom  Angelo  (La  Trobe  University),  Emer.  Prof   Stuart  Campbell  (University  of  Western  Sydney),  A/Prof  Martin  Carroll  (Charles  Darwin  University),  Prof  Liz  Deane  (formerly   Australian  National  University),  A/Prof  Duncan  Nulty  (Griffith  University),  Prof  Pip  Pattison  (The  University  of  Melbourne),   Prof  Belinda  Probert,  Prof  Judyth  Sachs  (Macquarie  University),  A/Prof  Ian  Solomonides  (Macquarie  University),  Prof  Suzi   Vaughan  (Queensland  University  of  Technology).  Project  Officer,  Ms  Kate  Aubin.   2  National  steering  group:  Emeritus  Professor  Christine  Ewan,  Dr  Peter  Ewell  (National  Council  for  Higher  Education   Management  Systems,  USA),  Professor  Mark  Freeman  (University  of  Sydney),  Mr  Ian  Hawke  (Tertiary  Education  Quality  and   Standards  Agency),  Professor  Royce  Sadler  (University  of  Queensland).  



CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information   Review  teams  first  evaluated  the  selected  unit’s  learning  outcomes,  its  learning  and  assessment   guide,  the  assessment  tasks  being  used,  and  the  marking  criteria  and  information  used  by  those   assessing  students’  work.  This  review  was  informed  by  contextual  information  provided  by  home   universities  regarding  course  learning  outcomes  and  structure.    In  this  way,  peer  reviewers  were   making  judgements  about  inputs,  or  teaching  standards,  by  evaluating  the  quality,  validity,  clarity,   relevance  and  standards  of  the  unit  inputs  from  a  partner  university.  Then,  using  the  assessment   criteria  and  marking  guides  provided  by  the  home  institution  responsible  for  delivering  the  unit,  two   partner  peer  reviewers  marked  a  selection  of  four  assessment  items  from  each  university,   representing  a  range  of  four  grade  bands  (Fail,  Pass,  Credit  and  Distinction/High  Distinction)  in  each   final  year  unit.  For  all  peer  reviewed  units,  universities  were  asked  to  provide  course/program  level   information  to  ensure  that  unit  outcomes  were  assessed  in  the  context  of  broader  course  level   outcomes.       The  project  generated  several  beneficial  outcomes  that  indicate  the  potential  for  scale  up  of  this   approach.  The  academics  who  participated  in  the  review  teams  reported  a  noteworthy  range  of   benefits.  For  example,  they  reported  that  the  approach  had  given  them  ‘an  idea  of  what  peers  in   other  universities  were  doing  in  similar  units’,  often  for  the  first  time.  It  also  provided  ‘reassurance   that  grading  standards  are  equivalent’.  Where  feedback  had  been  provided  by  peer  review  teams  on   potential  areas  for  improving  the  validity  and  quality  of  assessment  inputs  or  grading,  participants   consistently  reported  that  they  found  this  situated  feedback  ‘very  helpful’  for  it  indicated  where  they   may  need  to  engage  further  in  departmental  and  inter-­‐university  calibration  of  assessment  inputs,   marking  and  consensus  moderation  activities.  Throughout  the  project,  emphasis  was  given  to  the   importance  of  assessing  unit  inputs  and  outcomes  in  the  broader  course  context.  In  some  cases,   participants  were  uncertain  about  where  to  find  relevant  course-­‐level  information  to  share  with  peer   reviewers  as  a  way  to  contextualise  the  unit  materials.    This  outcome  highlights  the  ongoing  need  to   ensure  that  unit  objectives,  learning  activities,  assessment  and  outcomes  are  considered  within  the   broader  course  or  program  context.       For  both  individual  institutions  and  the  sector  overall,  the  blind  peer  review  methodology  was  seen   to  provide  a  feasible  and  objective  way  to  assure  and  improve  both  learning  outcomes  and   assessment  standards,  especially  if  a  sampling  approach  was  adopted,  involving  review  of  different   groups  of  selected  core  units  within  a  particular  discipline  area  each  year.  The  inclusion  of  the  full   range  of  university  types  in  this  study  was  seen  by  participants  to  be  particularly  advantageous  for  it   provided  an  opportunity  for  cross-­‐sectional  analysis  of  teaching  and  learning  standards  –  something   that  may  not  necessarily  be  available  when  peer  review  is  limited  to  particular  university  types  or   consortia.       While  there  was  broad  agreement  among  peer  reviewers  from  a  range  of  university  types  on  the   teaching  and  learning  standards,  as  defined  in  this  project  and  within  the  units  of  study  reviewed,     agreement  was  most  evident  in  judgements  about  threshold  Pass/Fail  grades.  It  was  suggested  by   participants,  as  well  as  the  project’s  management  team  and  national  steering  group,  that  the     methodology  used  in  this  study  is  a  useful  way  in  which  to  report  publicly  on  the  comparability  of   assessment  inputs  and  outcomes  -­‐  or  learning  standards  –  across  Australia,  whilst  avoiding  a  one-­‐ size-­‐fits-­‐all  approach  to  monitoring  and  assuring  standards.  The  process  was  seen  to  help  maintain   sector  diversity  within  some  general  parameters  and  to  be  feasible,  especially  if  a  sampling  approach   to  moderation  and  peer  review  is  adopted.  It  was  seen  as  having  the  potential  to  foster  a  process  of   continuous  improvement  in  the  quality  and  comparability  of  the  sector’s  learning  standards,  not  just    


CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information   to  assure  them.  It  can  do  this,  said  participants,  by  first  helping  each  participating  institution  identify   situated  gaps  in  assessment  quality  and  then,  most  importantly,  to  identify  potentially  relevant   improvement  solutions  by  looking  at  what  partner  institutions  do  in  equivalent  units  of  study,   discipline  areas  and  degree  programs.  Assessment  improvement  actions  may  include  reviews  of   policy  and  practice  in  relation  to  unit  and  course  design,  assessment,  academic  staff  calibration   activities,  including  for  sessional  staff,  and  consensus  moderation  practices.     Reporting  at  a  national  level  could  involve  the  identification  of  a  suite  of  final  year  undergraduate   units  of  study  for  inter-­‐institutional  peer  review  in  a  given  year.  Due  consideration  would  need  to  be   given  to  the  role  of  other  processes,  including  institutional  review  mechanisms  and  external   accreditation  frameworks.  Nevertheless,  a  national  collaborative  approach  to  inter-­‐institutional  peer   review  of  teaching  and  learning  standards,  focussing  on  the  validity  and  reliability  of  assessment,   would  make  a  significant  contribution  to  monitoring,  assuring  and  ongoing  improvement  of  quality  in   learning  and  teaching.  The  potential  for  the  Office  for  Learning  and  Teaching  to  play  a  role  in   identifying  and  disseminating  best  practice  in  assessment  in  different  disciplines,  programs/courses   and  units  of  study  as  a  further  resource  for  improvement  action  is  addressed  in  the  project   recommendations  to  follow.      

Project  deliverables  include:     i. a  validated  inter-­‐institutional  peer  review  model  that  is  feasible  and  scaleable  for  use  across  the   sector;     ii. a  user  guide  and  ready-­‐to-­‐use  resources  for  academic  staff  and  senior  university  leaders  to   enable  self-­‐regulated  accountability  and  academic  peer  review  and  reporting  mechanisms  that   are  transparent,  rigorous  and  sustainable;   iii. establishment  of  the  relative  benefits  of  a  range  of  approaches  to  the  use  of  disciplinary  peer   review  for  monitoring  and  assuring  standards;     iv. identification  and  user  testing  of  a  distinctive,  situated  and  flexible  approach  to  academic  staff   professional  development  through  sharing  of  the  assessment  and  grading  practices  of   colleagues  teaching  final  year  units  in  the  same  discipline,  including  sharing  of  course  learning   outcomes,  unit  outlines  and  grading  guides;     v. a  review  of  the  language  associated  with  peer  review  of  standards  and  recommendations   regarding  the  establishment  of  a  common,  sector-­‐wide  terminology  for  use  in  both  self-­‐ accrediting  and  non  self-­‐accrediting  higher  education  providers;   vi. a  project  website  which  enables  interested  institutions  to  quickly  and  conveniently  access  the   resources  and  information  produced  by  the  project:       In  addition  to  the  outcomes  specified  at  the  outset  of  the  project,  the  impact  of  the  project  can  be   evidenced  by  take  up  of  the  methodology  by  institutions  outside  the  project  team,  as  well   embedding  of  a  version  of  a  peer  review  and  moderation  process  in  university  policies.       To  download  the  User  Guide  and  Peer  Feedback  Forms  for  implementing  inter-­‐institutional  peer   review  of  standards  visit  .  Templates  are  provided  and  may  be   adapted  according  to  the  purpose  and  context  of  the  peer  review  activity.          



CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information  

2 Recommendations    

Based  on  the  findings  and  outcomes  of  this  project,  it  is  recommended  that:   1. The  Chair  of  Australia’s  Higher  Education  Standards  Panel  advocate  for  the  use  of  the   project’s  ‘blind’  peer  review  methodology  as  a  means  to  monitor  and  assure  both  objectively   and  efficiently  the  quality  and  comparability  of  disciplinary  learning  and  assessment   standards  across  Australia’s  HE  system.   2. TEQSA  endorse  the  process  tested  in  this  study  as  an  efficient  and  effective  way  in  which  to   externally  assure  the  assessment  standards  of  Australian  higher  education  at  the  disciplinary,   subject  and,  over  time,  the  institutional  level;  and  further,  that  this  endorsement  apply  to   both  self-­‐accrediting  and  non  self-­‐accrediting  institutions  of  higher  education.   3. The  Office  for  Learning  and  Teaching  create  a  one  to  two  year  secondment  position  for  an   expert  in  assessment  and  calibration  as  a  National  Assessment  Quality  and  Standards   Fellow/Advisor  to   a. assist  the  higher  education  sector  to  establish  the  policy  and  practice  frameworks  to   embed  inter-­‐institutional  peer  review  of  teaching  and  learning  standards  and     b. identify  and  disseminate  the  most  effective  assessment  practices  identified  through   peer  review  in  each  professional  or  disciplinary  area.   4. Higher  education  providers  collaborate  with  peak  disciplinary  and  professional  bodies,  under   the  coordination  of  the  proposed  OLT  Advisor/Fellow  (see  recommendation  3)  to  ensure  that   academic  staff  develop  skills  in  articulating  and  using  the  full  range  of  reference  points  now   available  for  the  purpose  of  monitoring  and  assuring  learning  outcome  standards  and   relevance.     5. Higher  education  providers  ensure  that  their  assessment  policies  and  quality  frameworks  are   reviewed  to  include  requirements  for  regular  inter-­‐institutional  peer  review  of  standards  for   final  year  undergraduate  units,  including  review  of  unit  inputs  and  assessment;  and  further   that  priority  be  given  to  ensuring  that  academic  staff,  including  sessional  staff,  have   appropriate  professional  development  to  normalise  consensus  moderation  and  calibration   activities  within  academic  departments.     6. Higher  education  providers  review  academic  workload  policies  and  role  statements  for   academic  staff  with  teaching,  coordination  and  assessment  responsibilities  to  reflect   expectations  regarding  regular  involvement  in  consensus  moderation  and  calibration   activities.   7. The  project  team  liaise  with  the  Office  for  Learning  and  Teaching  and  other  organisations   such  as  ACER  to  discuss  establishing  a  national  clearing  house  of  validated  assessment  items   by  discipline,  based  on  positive  ratings  from  peer  reviewers  involved  in  peer  review  activities   across  the  sector.     8. Further  investigation  be  undertaken  to  determine  if  the  use  of  final  year  capstone  units  of   study  and  assessment  tasks  are  a  valid  and  feasible  way  in  which  to  evaluate  graduating   students’  capability  to  integrate  and  appropriately  apply  what  they  have  learnt  in  individual   units  of  study  to  addressing  real  world  dilemmas  and  challenges  in  their  selected  discipline  or   profession  effectively. ========================================  



OLT_LaTS summary_Criterion Wkshop - Western Sydney University

CONSULTATION  PAPER   please  contact  Kerri-­‐Lee  Krause  ([email protected])  for  further  information         Not  for  further  publication  ...

147KB Sizes 0 Downloads 0 Views

Recommend Documents

HIE | Publications - Western Sydney University
Ellsworth DS, Anderson IC, Crous KY, Cooke J, Drake JE, Gherlenda AN, Gimeno TE, Macdonald CA, Medlyn BE, Powell JR, Tjo

Mail Merge - Western Sydney University
Mail Merge .....Rather than typing the same information repeatedly you can set up a mail merge between the Student Acade

Child Friendly Bolivia - Western Sydney University
About 63% of the Bolivian population lives in urban areas; of this group, 61% lives in slums. Children ..... Metodologí

Reclaiming Australian Multiculturalism - Western Sydney University
Sep 8, 2016 - 'Reclaiming Australian Multiculturalism: policy and practice in a .... The initial concept of multicultura

Yiye LU - ResearchDirect - University of Western Sydney
May 27, 2014 - materials for Chinese learning, specifically for the English-speaking leaners. The. Character films could

ASEARC Proceedings - University of Western Sydney
These proceedings contain the papers of the Fourth Applied Statistics Education and Research Col- laboration (ASEARC) Co

Aesthetics, Government, Freedom - University of Western Sydney
The journal Key Words: A Journal of Cultural Materialism is available online: ..... In this way, in these terms, he inve

Download full program - Western Sydney University
Nov 15, 2017 - Palacio del Vino – Seafood/South American/Wine bar. Avenida Brasil #75. Concha y Toro 42 - Chilean/Sout

Harvard Referencing Style Guide - Western Sydney University Library

excel functions – full list - Western Sydney University
EXCEL FUNCTIONS – FULL LIST. This is a complete alphabetical list of all the Functions in Excel with a description. To

1x938 Acacias 38 | 時間停止21 | Nonton Movie