Outcomes Assessment:
|
Report of Taskforce on Quality Assurance June 9, 1994 Members:
I. Executive Summary The legislature, CSU system, and accreditation standards all call for evidence of the effectiveness and outcomes of academic programs and the institution as a whole. This committee studied the current level of assessment activities on campus, reviewed the growing literature on outcomes assessment, and developed a plan to phase in a program. The plan calls for each department and program to 1) prepare a mission statement, 2) set forth goals and objectives, 3) establish criteria to be measured and reported, and 4) develop a method and timetable for evaluation of the results. The overall purpose of these departmental assessment plans is to improve program quality and to assure that students meet the level of competency and accomplishments specified in the goals and objectives. The 1994-95 academic year would be devoted to development of the individual departmental plans and testing of a selected number of techniques and methods through pilot projects. General Education and Basic Skills Assessment is a much more critical and complicated topic; an assessment program will be more time consuming and expensive to implement in these areas. The Committee recommends that additional study and planning be undertaken to experiment with various means of assuring minimal competency in reading, writing, quantitative/problem solving skills and critical thinking. Additionally, departments would be asked to examine their curricula to determine how and where general education skills would be strengthened in their majors. Development of baseline data, collection of evidence of student growth and value-added by the program, and certification of exit competency are needed as well as some comparison of our students to national norms and standards. II. Charge of the Taskforce: The Committee was appointed by Vice President Hefner in Fall of 1993 to examine our current efforts in quality assurance and to develop a plan for campus-wide involvement in outcomes assessment activities. The overall goal was to insure the quality and effectiveness of our programs. Increasing demands are being placed upon higher education to document student achievement. The national trend has been towards defining those outcomes expected of all college graduates from both the major and general education, and then implementing programs to assure that the curriculum is designed to both teach and reinforce those outcomes. Although, CSU San Bernardino is engaged in a number of valuative/assessment activities, including basic skills testing (ELM, EPT), specialized accreditation of professional programs, five-year program reviews, surveys of our recent graduates, evaluation of specific programs (SAIL, EOP, Freshman Seminar), General Education Reviews and our campus-wide WASC accreditation, there presently is no systematic evaluation of our overall performance. Our assessment efforts tend to focus on specific components in isolation and usually do not address the totality of our curriculum or the competency level of students at graduation. Soon after their initial appointment, six members of the Taskforce attended a workshop by the Association of American College on Reforming and Assessing the Academic Major. This was part of a multi-year follow up study and dissemination of successful strategies of pilot projects from institutions around the country who were attempting to carry out the goals and criteria developed in the organization's studies Integrity in the College Curriculum and Reports from the Field, which attempt to establish common core curricula for General Education and liberal arts majors. The growing literature on outcomes assessment was examined along with sample material from other campuses, notably Ball State University, the University of Charleston, and the University of Arkansas, Little Rock. Meetings were held with Deans and department chairs to answer questions, discover the current state of evaluation and assessment of program effectiveness in departments, and to share material employed at other institutions and under consideration by the Taskforce. (See attached inventory of departmental assessment programs at CSUSB.) Prior Efforts Most of our prior assessment studies have been conducted through periodic surveys of current students, recent alumni, employers, and other relevant community groups. These studies have been conducted at several levels as described below. University level The Office of Institutional Research collects and analyzes student enrollment and course data each quarter. This data is used to generate reports related to student demographics, enrollment data, and FTES. This data is also used to identify student majors and track continuing students by program. The Graduate Studies Office conducts an annual survey of Graduate Alumni. This survey is used to determine student satisfaction with graduate studies at CSUSB. It also tracks employment of graduates and admission to other graduate degree granting institutions by program. School level The Office of Institutional Research works with each School to collect data needed for the Self-Study and program review required by the Chancellor's Office every five years. Recent alumni from each school are surveyed during the winter term and currently enrolled students are surveyed during the spring term preceding each School's Self-Study. Each School may also conduct additional student, alumni and employer surveys to satisfy external accreditation bodies. The WASC Study and the recent accreditation of the Business School are examples of such data collection efforts. The School of Education is required to conduct annual follow-up studies on persons receiving basic or advanced credentials through the School of Education. Department/Program level Many individual Departments or Programs conduct periodic surveys of current students and alumni to satisfy the requirements of external accrediting bodies. The Social Work program and the Rehabilitation Counseling program are examples of recently accredited programs that were required to conduct extensive need assessments and collect current student and alumni data. Many individual departments and programs assess the knowledge and skill level of their current students through a variety of assessment techniques. These include objective and subjective examinations, portfolio assessment, clinical supervision and practicum, capstone courses, and student interviews. Final assessment of student learning is assessed through comprehensive examinations, thesis, directed projects, performance/recital, or performance on state or national norm referenced examinations. Campus-wide Issues The committee also examined the state of assessment and quality assurance programs throughout the institution using a model developed by the Southern Association of Schools and Colleges. This inventory of assessment practices found many areas within the University which need to develop assessment activities. (See attached inventory.) III. General Principles Guiding the Committee and Report 1. Each department and program should be expected to develop and implement a program of outcomes measurement. 2. The current Faculty Senate policy document on program review procedures and guidelines should be revised to focus on outcomes measurement and demonstration of proof of the effectiveness of the program in reaching its stated goals and objectives. 3. Quality assurance and program effectiveness for academic programs cannot be done in isolation from a general concern for quality in other areas of the institution. The institution should develop a cycle for review of program effectiveness for each program, office and division in the campus. This review of quality needs to be done in conjunction with an overall structure for University planning, budgeting, evaluation and program revision. 4. The general change from an emphasis on "input variables" (resources, selectivity, space, faculty, course work completed) to examination of the results and outcomes of degree programs (skills attained, objectives reached, graduation rate, employer satisfaction, etc.) represents a major shift in higher education. Therefore, it must be approached carefully and implemented without haste. Program reviews and assessment plans should be phased-in where possible. 5. The academic departments (and institution) should concentrate first on issues and areas which can be funded under current budget restrictions and which rely primarily on available data and existing procedure if possible. Elaborate and costly programs of outcomes assessment should not be developed at this time. It should be recognized, however, that assessment and program evaluation must be a campus priority and outcomes data should be used in the budget process and value system of the university for any assessment program to succeed. 6. The costs of assessment should not be borne exclusively by the students through imposition of high fees. Quality assurance is a basic function of each program. If standardized, commercially available tests are to be used for sampling and comparison of groups of students with national norms, this should be supported by the university if such measures are approved as part of the program's assessment plan. 7. Multiple measures and means of assessment should be employed. One methodology will not fit all disciplines or all students. 8. Criteria and measurement instruments should be developed by faculty--not an outside committee. Certain baseline data on enrollment, retention and graduation rates, placement rates, surveys of student, graduates and employers should be regularly collected by Institutional Research to assist departments. This needs to be standardized in format, timing and data elements to be collected and used. 9. During the early phases, assessment efforts should concentrate on program effectiveness and aggregate data--not individual students. Development of minimal competency standards which all students are expected to meet should be encouraged, however, especially in key areas of basic skills and general education knowledge. 10. An ongoing committee structure needs to be established to guide, assist and evaluate assessment efforts. For example, someone needs to decide if departmental criteria and procedures are adequate and to evaluate the program effectiveness program over time. In the early stages of developing and implementing program assessments, this committee would provide expertise and perspective to the schools and departments. 11. Programs which must meet specialized accreditation or licensure standards may find a conflict between campus standards and/or program review and assessment procedures. Departmental plans may require variance and flexibility to adapt to the situation. Local objectives and standards may be either more or less demanding or specific. Time lines and review cycles also may be in conflict. IV. Components of Proposed Program Effectiveness Plan Each department and program at a minimum should be required to develop the following: Mission Statement setting forth the aspirations and intentions of the program Goals: A set of Circumstances to be sought in pursuit of the mission Objectives: Specific ends to be achieved with regard to a particular goal Criteria and list of key elements to be used in assessing student and program outcomes A Methodology and Time line for carrying out the assessment An Evaluation Plan for use of information obtained in the Assessment as part of program review and improvement The Components of the Mission Statement should cover broad philosophical goals and directions. This would include the aspirations and intentions of the program: "Where is the program now? and where do they want it to be?" Other aspects that might be included: The Intellectual Rationale and Philosophy of the program Fundamental Purpose of department/program. Role and Scope Basic Assumptions and Values of the program Areas of Emphasis or Excellence. Unique Attributes. Priorities to be stressed Overall Goals of the program and curriculum General Outcomes students are to be prepared for Types of students and societal needs to be addressed Relationship of the Program Mission to General Education Program, Mission of School and Institution's Mission Statement It should be noted that these mission statements will be evolving over time and that the assessment and evaluation of progress will often lead to reexamination of mission. Development of Goals and Objectives: Goals and Objectives are similar in that they serve to direct teaching and learning. They describe the intended purposes and expected results of teaching activities and establish the foundation for assessment. "Goals" Defined: Goals are statements about general aims or purposes of education that are broad, long-range intended outcomes. Goals are primarily used in policy making and general program planning. Examples of Goal Statements: -- To train students in methods of scholarly inquiry, scientific research, and problems solving -- To create within students an appreciation of the uniqueness of the sociological perspective and of its value in the explanation of real-world issues -- To have students participate in out-of-class activities relevant to anthropology -- To ensure that all students read, write and speak effectively -- To help students acquire depth of knowledge and competence in an academic discipline "Objectives" Defined: Objectives are brief, clear statements that describe learning outcomes of instruction. Attention is focused on the specific types of performances that students are expected to demonstrate at the end of instruction. Objectives are the intended results; Outcomes are the achieved results. A statement of objective should ideally precede assessment. "One must know what is to be assessed before one knows how to assess it." In many cases, however, goals and objectives were predetermined or implied and assessment has been ongoing without explicit or written or commonly agreed upon objectives. Objectives may be classified as cognitive, affective, or skill/performance dimension. Cognitive objectives are descriptions of thinking skills. Affective objectives refer to attitudinal, personal, and social dimension of outcomes. Skill objectives such as writing, computer use, speaking, and physical skills are distinctive from cognitive (knowledge) objectives. Skills are means by which knowledge is acquired. Subject matter objectives are a subset of cognitive objectives that are specific to knowledge of subject matter. One can also differentiate between Mastery and Developmental Objectives: Simple knowledge and skills and advanced or higher level of learning are very distinctive. Mastery objectives are typically concerned with the minimum performance essentials--those learning tasks that must be mastered by all students for success at the next level of instruction. These objectives tend to be limited enough in scope that all, or nearly all, intended outcomes can be specified.
Examples: Developmental objectives are concerned with more complex learning outcomes--those learning tasks toward which students can be expected to show varying degrees of progress. Developmental objectives are often written in a two-step process in which a general objective is stated along with a sample of specific learning outcomes.
Examples of Program Objectives: Students will learn skills in formal and informal reasoning Seniors in the major will compare favorably with other students in psychology in the nation on a general test of psychological knowledge Students will demonstrate knowledge of representative authors, themes, and genres of British and American Literature Students will write Spanish with: accurate grammar including correct use of most important verb tenses; vocabulary appropriate to a variety of functional and social situations Students will demonstrate basic music literacy in reading and writing music Examples of Assessment Criteria and Procedures: Senior recitals with audiotapes of performance are regularly evaluated by faculty and outside assessors Seniors will complete a technical writing report on a senior independent research project that exemplifies the level of competence set by the department The Psychology Department will administer the Major Field Achievement Test of the Educational Testing Service Writing Sample Analysis of Student Writing from Spring 1994 set of courses will evaluate use of the department's standards for Philosophy Student Writing Guide Oral examination on Senior Paper will indicate majors' ability to write and speak effectively Exit Survey with graduating seniors will assess majors' sense of accomplishment of objectives for the major Portfolio of student papers will demonstrate overall performance of major in relation to stated objects/outcomes of major V. General Education Outcomes Assessment This area is very problematic and also will be the most costly and most difficult to implement. It will take additional time to develop a full assessment plan. The following steps are suggested: 1. The current general education curriculum should be reviewed. The General Education Committee has completed a review of three of the four broad sections of general education and will complete the fourth next year. They have called for revision of the current Senate policy in order to have a review of the total package during the fifth year. Use of outside consultants is recommended. 2. The criteria for each of the GE categories needs to be reexamined for currency and adequacy. The criteria should be expanded to include specific outcomes expected as well as the curriculum content to be delivered. Departments should be asked to provide evidence that their courses provide the knowledge, skills and outcomes expected. 3. Baseline data on skill levels of entering and graduating students should be obtained in the areas of reading comprehension, writing, critical thinking, and quantitative/problems solving skills. (Time line: Fall and Winter of 1994-95) 4. Transfer students should be examined in comparison to native freshmen and graduates. (Time line: Fall of 1994.) Transcript analysis is needed to determine the percentage completing Basic Skills prior to coming to CSUSB and the amount and percentage of GE completed at previous institutions. This is a critical topic as so many of our students are transfers. The issue of "leveling" is particularly thorny because we are required by law and public policy to accept certified transfer work. 5. Use of nationally norm-referenced tests of Basic Skills and General Education knowledge (such as ACT's College Outcomes Measurement Program or Educational Testing Service's Academic Profile or Riverside Publishing's College BASE) is recommended for at least a sample of incoming and graduating students. 6. Departments should be asked to analyze their courses to indicate what portions of the GE program they will supplement and build on in their major. (Time line: March of 1995) 7. Committees should be established in each of the basic skills areas to define goals and outcomes for reading, writing, critical thinking and mathematics and to develop some method of assuring that all students have mastery of these vital skill areas. (These studies should provide input to the General Education Review and Revision suggested above.) |