1 Workshop on Improving Education Deliverance and Attainment Standards Through Transforming Academic Institutions Towards OBE System Megat Johari Megat Mohd Noor Professor, Malaysia Japan International Institute of Technology & Assoc Director (International Affairs), Engineering Accreditation Department Karachi & Peshawar, Pakistan October 2015

2 Programme Time Day 2 09.00 – 10.30 Evaluating Programme I10.30 – 10.45 Tea 10.45 – 13.00 Evaluating Programme II 13.00 – 14.00 Lunch & Zuhr Prayer 14.00 – 15.30 Complex Problem Solving I 15.30 – 15.45 15.45 – 16.45 Complex Problem Solving II 16.45 – 17.00 Closing Remarks & Tea

3 Outlines Introduction Taxonomy Programme Outcomes Knowledge ProfileLevel of Problem Solving Exemplars Conclusion

4 Challenges Paradigm Shift – Outcome & QualityMaintain Fundamentals while Encourage Inclusion of Latest Technology Advancement in the Curriculum Allow Academic Innovation and Creativity Avoid Side-tracked Variety of Modes of Delivery

5 Engineering & Technology DomainEngineers Career in Research & Design Career in Supervision & Maintenance Work Technologists Strong in Mathematics, Engineering Sciences, Professional courses (Theoretical) Appropriate Mathematics, Engineering Sciences, Professional courses (Practical) Education Engineering Breadth & Depth of Curricula Technology Breadth & Depth of Curricula

6 Expectations of AccreditationEducation content and level (depth) are maintained Programme Continual Quality Improvement (CQI) Outcome-based Education (OBE) Programme Systematic (QMS)

7 PEO & PO Students FacilitiesStaff QMS Facilities Curriculum CQI CRITERIA

8 Different Levels of OutcomesProgramme Educational Objectives Few years after Graduation – 3 to 5 years Programme Outcomes Upon graduation Course/subject Outcomes Upon subject completion Weekly/Topic Outcomes Upon weekly/topic completion

9 Outcome-Based AssessmentImplementation Strategy Assessment Strategy Data Sources/Assessment instruments Industrial project Improve student competence in communication, teamwork, and project management Exams, interview, survey, observe, assess skill level, monitor development of skills Reports, interview schedule, survey, observation records, grades of exams and projects, exit skill checklist Design course Address industry needs Assessment by industry, and lecturers List of assessment criteria, observation, reports, interview, students evaluation, exams, exit skill checklist

10 Big Picture Assessment – Constructive AlignmentProgramme or Student Improvement ? PHILOSOPHY ? Design Selective Culminating Hybrid MODEL ? Attainment Taxonomy Level (Average, From, Up To)

11 Programme Objectives What is expected (3-5 years) upon graduation (What the programme is preparing graduates in their career and professional accomplishments)

12 Programme Outcomes What the graduates are expected to know and able to perform or attain by the time of graduation (knowledge, skills/psychomotor, and affective/interpersonal/attitude) There must be a clear linkage between Objectives and Outcomes Need to distribute the outcomes throughout the programme, and not one/two courses only addressing a particular outcome

13 PO Attainment Knowledge Skills Affective Knowledge Skills AffectiveFinal Year Project Final Year Project Final Year Design Project Final Year Design Project Final Year Courses Final Year Courses Knowledge Skills Affective Third Year Courses Third Year Courses Second Year Courses Second Year Courses Knowledge Skills Affective First Year Courses First Year Courses

14 Compliance to Washington AccordKnowledge Profile Level of Problem Solving Graduate Attributes (Programme Outcomes)

15 ENGINEERING KNOWLEDGE UNIVERSITY EXPERIENCEPEO WHAT YOU WANT YOUR GRADUATES TO BE IN YEARS EXTRA-CURRICULAR WA 1 ENGINEERING KNOWLEDGE WA 2 PROBLEM ANALYSIS 4 YEARS WA3 DESIGN WA9 IND & TEAM UNIVERSITY EXPERIENCE WA5 MODERN TOOLS WA10 COMMUNICAT-ION WA6 ENGR & SOC WA7 ENV & SUST WA8 ETHICS WA11 PROJ MGMT & FINANCE WA4 INVESTIGATION WA12 LIFE LONG

16 Course Outcomes Statement … explain, calculate, derive, design, critique. Statement … learn, know, understand, appreciate – not learning objectives but may qualify as outcomes (non-observable). Understanding cannot be directly observed, student must do something observable to demonstrate his/her understanding.

17 lower order Intermediate Higher orderEAC Training Modules (OBE for Panel Evaluator)

18 lower order Intermediate Higher orderEAC Training Modules (OBE for Panel Evaluator)

19 lower order Intermediate Higher orderEAC Training Modules (OBE for Panel Evaluator)

20 New Bloom’s Taxonomy Bloom’s Taxonomy Knowledge (list)Comprehension (explain) Application (calculate, solve, determine) Analysis (classify, predict, model,derived) Synthesis (design, improve) Evaluation (judge, select, critique)

21

22 Three components of a learning outcomeVerb (V), Condition (C) & Standard (S) describe the principles used in designing X.(V) orally describe the principles used in designing X. (V&C) orally describe the five principles used in designing X. (V&C&S) design a beam. (V) design a beam using Microsoft Excel design template . (V&C) design a beam using Microsoft Excel design template based on BS 5950:Part 1. (V&C&S)

23 Learning outcomes by adding a condition and standardPoor Students are able to design research. Better Students are able to independently design and carry out experimental and correlational research. Best Students are able to independently design and carry out experimental and correlational research that yields valid results. Source: Bergen, R A Program Guideline for Outcomes Assessment at Geneva College

24 Learning Style Model Perception Sensing IntuitiveInput Modality Visual Verbal Processing Active Reflective Understanding Sequential Global

25 Problem Organised Project Work or POPBL (Project Oriented Problem Based Learning)Group Studies Lectures Literature Problem Analysis Problem Solving Report Tutorials Field Work Experiment

26 Depth of Knowledge RequiredComplex Problems (Engineer) Broadly Defined Problems (Technologist) Well defined Problems (Technician) Requires in-depth knowledge that allows a fundamentals-based first principles analytical approach Requires knowledge of principles and applied procedures or methodologies Can be solved using limited theoretical knowledge, but normally requires extensive practical knowledge

27 Washington Accord Graduate Attributes PROGRAMME OUTCOMESEngineering Knowledge Breadth & depth of knowledge WA2 Problem Analysis Complexity of analysis WA3 Design/Development of Solutions Breadth & uniqueness of engineering problems i.e. the extent to which problems are original and to which solutions have previously been identified and coded WA4 Investigation Breadth & depth of investigation and experimentation WA5 Modern Tool Usage Level of understanding of the appropriateness of the tool WA6 The Engineer and Society Level of knowledge and responsibility WA7 Environment and Sustainability Type of solutions WA8 Ethics Understanding and level of practice WA9 Individual and Team Work Role in and diversity of team WA10 Communication Level of communication according to type of activities performed WA11 Project Management and Finance Level of management required for differing types of activity WA12 Life-long Learning Preparation for and depth of continuing learning

28 PROGRAMME OUTCOME (i) Engineering Knowledge(WA1) Apply knowledge of mathematics, natural science, engineering fundamentals and an engineering specialisation to the solution of complex engineering problems; (WK1 to WK4)

29 PROGRAMME OUTCOME (ii) Problem Analysis - Complexity of analysis(WA2) Identify, formulate, research literature and analyse complex engineering problems reaching substantiated conclusions using first principles of mathematics, natural sciences and engineering sciences (WK1 – WK4)

30 PROGRAMME OUTCOME (iii) Design/Development of Solutions – Breadth and uniqueness of engineering problems i.e. the extent to which problems are original and to which solutions have previously been identified or codified (WA3) Design solutions for complex engineering problems and design systems, components or processes that meet specified needs with appropriate consideration for public health and safety, cultural, societal, and environmental considerations (WK5)

31 PROGRAMME OUTCOME (iv) Investigation - Breadth & Depth of Investigation & Experimentation (WA4) Conduct investigation of complex problems using research based knowledge (WK8) and research methods including design of experiments, analysis and interpretation of data, and synthesis of information to provide valid conclusions

32 PROGRAMME OUTCOME (v) Modern Tool Usage - Level of understanding of the appropriateness of the tool (WA5) Create, select and apply appropriate techniques, resources, and modern engineering and IT tools, including prediction and modelling, to complex engineering problems, with an understanding of the limitations. (WK6)

33 PROGRAMME OUTCOME (vi) The Engineer and Society - Level of knowledge and responsibility (WA6) Apply reasoning informed by contextual knowledge to assess societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to professional engineering practice and solutions to complex engineering problems. (WK7)

34 PROGRAMME OUTCOME (vii) Environment and Sustainability - Type of solutions (WA7) Understand and evaluate the sustainabilty and impact of professional engineering work in the solutions of complex engineering problems in societal and environmental contexts (demonstrate knowledge of and need for sustainable development) (WK7)

35 PROGRAMME OUTCOME (viii) Ethics - Understanding and level of practice(WA8) Apply ethical principles and commit to professional ethics and responsibilities and norms of engineering practice. (WK7)

36 PROGRAMME OUTCOME (x) Individual and Team Work – Role in and diversity of team (WA9) Function effectively as an individual, and as a member or leader in diverse teams and in multi-disciplinary settings

37 PROGRAMME OUTCOME (ix) Communication – Level of communication according to type of activities performed (WA10) Communicate effectively on complex engineering activities with the engineering community and with society at large, such as being able to comprehend and write effective reports and design documentation, make effective presentations, and give and receive clear instructions

38 PROGRAMME OUTCOME (xi) Project Management and Finance – Level of management required for differing types of activity (WA11) Demonstrate knowledge and understanding of engineering and management principles and economic decision-making and apply these to one’s own work, as a member and leader in a team, to manage projects and in multidisciplinary environments

39 PROGRAMME OUTCOME (xii) Life-long Learning – Preparation for and depth of continuing learning (WA12) Recognise the need for, and have the preparation and ability to engage in independent and life-long learning in the broadest context of technological change

40 Knowledge Profile (Curriculum)Theory-based natural sciences WK1 Conceptually-based mathematics, numerical analysis, statistics and formal aspects of computer and information science to support analysis and modelling WK2 Theory-based engineering fundamentals WK3 Engineering specialist knowledge that provides theoretical frameworks and bodies of knowledge for the practice areas; much is forefront WK4

41 Knowledge Profile Knowledge that supports Engineering design in the practice areas WK5 Knowledge of Engineering practice (technology) in the practice areas WK6 Comprehension of the role of Engineering in society and identified issues in engineering practice: ethics and professional responsibility of an engineer to public safety; the impact of engineering activity: economic, social, cultural, environmental and sustainability WK7 Engagement with selected knowledge in the Research literature WK8

42 Knowledge Profile 4 YEARS WK1 WK5 natural sciences engineering designmathematics, numerical analysis, statistics, computer and information science 4 YEARS WK6 engineering practice WK7 engineering in society WK3 engineering fundamentals WK8 research literature WK4 engineering specialist knowledge

43 4 YEARS WA9 WA3 IND & TEAM DESIGN WA1 ENGINEERING KNOWLEDGE WA2PROBLEM ANALYSIS 4 YEARS WK1 natural sciences WA9 IND & TEAM WK5 engineering design WA3 DESIGN WK2 mathematics, numerical analysis, statistics, computer and information science WA10 COMMUNICAT-ION WK6 engineering practice WA5 MODERN TOOLS WK7 engineering in society WA6 ENGR & SOC WA7 ENV & SUST WA8 ETHICS WA11 PROJ MGMT & FINANCE WK3 engineering fundamentals WK8 research literature WA4 INVESTIGATION WA12 LIFE LONG WK4 engineering specialist knowledge

44 4 YEARS WA9 WA3 IND & TEAM DESIGN WA1 ENGINEERING KNOWLEDGE WA2PROBLEM ANALYSIS 4 YEARS WK1 natural sciences WA9 IND & TEAM WK5 engineering design WA3 DESIGN WK2 mathematics, numerical analysis, statistics, computer and information science WA10 COMMUNICAT-ION WK6 engineering practice WA5 MODERN TOOLS WK7 engineering in society WA6 ENGR & SOC WA7 ENV & SUST WA8 ETHICS WA11 PROJ MGMT & FINANCE WK3 engineering fundamentals WK8 research literature WA4 INVESTIGATION WA12 LIFE LONG WK4 engineering specialist knowledge

45 Need to think broadly and systematically and see the big pictureComplex Problem Need to think broadly and systematically and see the big picture Complex Problem Difficult Decision Uncertain Strategy Confusing Idea Contentious Product Intractable Change

46 Difficulty & UncertaintyComplexity – the problem contains a large number of diverse, dynamic and interdependent elements Measurement – it is difficult or practically unfeasible to get good qualitative data Novelty – there is a new solution evolving or an innovative design is needed

47 Characteristics Technical Problems Complex ProblemsIsolatable boundable problem Universally similar type Stable and/or predictable problem parameters Multiple low-risk experiments are possible Limited set of alternative solutions Involve few or homogeneous stakeholders Single optimal and testable solutions Single optimal solution can be clearly recognised No definitive problem boundary Relatively unique or unprecedented Unstable and/or unpredictable problem parameters Multiple experiments are not possible No bounded set of alternative solutions Multiple stakeholders with different views or interest No single optimal and/or objectively testable solution No clear stopping point

48 Scientific/Technical Problemscan combine to form A Complex Problem

49 Complex Technical Limited Explanation, Prediction, ControlResults in an educated guest ? A limited number of features are captured by the Model Operating with scare resources Difficult to measure Complex causal Chains Unbounded Systems, No Experiment Explanation, Prediction, Control Results in a Covering Law f(x,y,z) All the Salient features are captured by the Model Operating with adequate resources Measurable Simple causal Chains Isolatable Systems, Controlled Experiment Complex Technical

50 Complex Problems (Need High Taxonomy Level)Complex Engineering Problems have characteristic WP1 and some or all of WP2 to WP7, EP1 and EP2, that can be resolved with in-depth forefront knowledge WP1 Depth of Knowledge required Resolved with forefront in-depth engineering knowledge (WK3, WK4, WK5, WK6 or WK8) which allows a fundamentals-based, first principles analytical approach WP2 Range of conflicting requirements Involve wide-ranging or conflicting technical, engineering and other issues. WP3 Depth of analysis required Have no obvious solution and require abstract thinking, originality in analysis to formulate suitable models. WP4 Familiarity of issues Involve infrequently encountered issues WP5 Extent of applicable codes Beyond codes of practice WP6 Extent of stakeholder involvement and level of conflicting requirements Involve diverse groups of stakeholders with widely varying needs. WP7 Interdependence Are high level problems including many component parts or sub-problems. EP1 Consequences Have significant consequences in a range of contexts. EP2 Judgement Require judgement in decision making

51 Complex Engineering Activities (Project based)Preamble Complex activities means (engineering) activities or projects that have some or all of the following characteristics listed below Range of resources Diverse resources (people, money, equipment, materials, information and technologies).EA1 Level of interaction Require resolution of significant problems arising from interactions between wide ranging or conflicting technical, engineering or other issues.EA2 Innovation Involve creative use of engineering principles and research-based knowledge in novel ways. EA3 Consequences to society and the environment Have significant consequences in a range of contexts, characterised by difficulty of prediction and mitigation.EA4 Familiarity Can extend beyond previous experiences by applying principles-based approaches.EA5

52 Problem Oriented, Team-Based Project Work as a Learning/Teaching DeviceProblem-oriented project-organized education deals with the solution of theoretical problems through the use of any relevant knowledge, whatever discipline the knowledge derives from. We are dealing with KNOW WHY (Research Problems). In design-oriented project work, the students deal with KNOW HOW problems that can be solved by theories and knowledge they have acquired in their previous lectures. (Design Problems).

53 Example 1: Complex Problem SolvingTwo villages in Timbuktu are separated from each other by a valley, at its deepest section about 30 metres. The valley is dry all the year around, except for the four months, from October to December each year, where torrential rainfall can flood major parts of the valley to a depth of over 12 metres in some site. The soil is generally lateritic with firm bedrock underneath. A bridge connecting the two villages is in a state of disrepair and has to be replaced. Write a project brief on how would you approach to design for the replacement bridge. You are limited to the use of locally available building materials. Heavy equipment is not available for the construction.

54 Aspects Economics Social Environment Ethics Management TechnologyAnalysis Evaluation

55 Thinking Site condition Weather Available technologyBuilding materials Design Costing Scheduling

56 Solutions? Problem solving skills Formulate the problem LiteratureExperiment?

57 Assessment Report – style and content (flow) Display – attractive ?Viva / Articulation Teamwork Management – scheduling

58 Example 2: Complex Problem SolvingRiver Spring Fissured Rocks Sandy soil Clayey soil Igneous rock Groundwater flow

59 How does complexity relates to curriculum?General Subjects Industrial Placement Core & Specialist (Engineering) Subjects – Complex Problem Solving Elective Subjects – Complex Problem Solving Design Project – Complex Problem Solving & Complex Engineering Activities Final Year Project – Complex Problem Solving

60 Establish, Maintain & Improve System Management CommitmentACCULTURALISATION QUALITY EDUCATION Knowledge Behaviour Attitude DNA Establish, Maintain & Improve System Resources Management Commitment

61 Conclusion Adequate knowledge profile Right taxonomyDemonstrate outcomes (solving complex problem)

62 Appendix

63 Complex Problem Solving (CPS)Dynamic, because early actions determine the environment in which subsequent decision must be made, and features of the task environment may change independently of the solver’s actions; Time- dependent, because decisions must be made at the correct moment in relation to environmental demands; Complex, in the sense that most variables are not related to each other in a one-to-one manner

64 Microworld CPS Model The problem requires not one decision, but a long series, in which early decisions condition later ones. For a task that is changing continuously, the same action can be definitive at moment t1 and useless at moment t2. Include novel solutions to an old dilemma in general science (external validity vs. experimental control)

65 Expert-novice CPS ModelExpert-novice approach most of the time produces conclusions that are crystal-clear. It almost guarantees statistically significant results, because the groups compared (expert and novices) are very different and tend to perform very differently when confronted with similar experimental situations (Sternberg 1995).

66 Naturalistic decision making (NDM)Naturalistic decision making (NDM) (e.g., Zsambok and Klein 1997, Salas and Klein 2001) ‘real-world’ task Example interviewing firefighters after putting out a fire or a surgeon after she has decided in real time what to do with a patient.

67 Dynamic decision making DDMDynamic decision making (DDM) (Brehmer 1992, Sterman 1994). Discrete dynamic decision tasks that change only when the participant introduces a new set of inputs. Variables like time pressure have been successfully integrated in models like Busemeyer and Townsend’s (1993) decision field theory

68 Implicit learning in system controlThis tradition has used tasks like the sugar factory (Berry and Broadbent 1984) or the transportation task (Broadbent et al. 1986), that are governed by comparatively simple equations. The theorization and computational modeling in this branch of CPS are extremely rich. Models are based on exemplar learning, rule learning, and both (e.g., Dienes and Fahey 1995, Gibson et al. 1997, Lebiere et al. 1998).

69 European complex problem solving (CPS)Initiated by Dörner (Dörner and Scholkopf 1991, Dörner and Wearing 1995) A large number of tasks that have been considered complex problem solving are nowadays affordable for theory development and computer modeling (e.g. Putz-Osterloh 1993, Vollmeyer et al. 1996, Burns and Vollmeyer 2002, Schoppek 2002) Transport real-life complexity to the lab in a way that can be partly controlled

70 Time related Time variant – time invariant (dynamic vs. static systems) Continuous time – discrete time. Degree of time pressure – decision has to be made quickly

71 Variable related Number and type (discrete/continuous) of variablesNumber and pattern of relationships between variables Non-Linear - Linear

72 System behaviour relatedOpaque - transparent. Stochastic - deterministic Delayed feedback - immediate feedback.

73 Delivery Knowledge-lean vs. knowledge-intensiveSkill based vs planning based (reactive vs predictive Learning vs. no learning during problem solving Understanding-based vs. search-based problems Ill-defined vs. well-defined

74 Conclusion Problem solving has been traditionally a task-centered field. VanLehn (1989) think that ‘task’ and ‘problem’ are virtually synonymous.

75