Focus Area A: Undertaking Systematic Planning and Evaluation that Includes Assessment of Student Learning Outcomes

The Commission on Institutions of Higher Education (CIHE) has focused our attention on making progress in undertaking planning, evaluation, and assessment:

We wish to see evidence, in the 2003 report, of the University's success in undertaking systematic planning and evaluation that includes the assessment of student learning outcomes. (Report of findings, October 19, 2000)

UMass Dartmouth needs to address two standards on Planning and Evaluation: 2.2, which stipulates, "Planning and evaluation are systematic, broad-based, interrelated, and appropriate to the institution's circumstances"; and 2.4, which stipulates that "...evaluation enables the institution to demonstrate through verifiable means its attainment of purposes and objectives both inside and outside the classroom."

3.1 Institutionalizing systematic institutional planning

This section identifies progress in institutionalizing systematic planning and then addresses future directions and challenges. UMass Dartmouth has made progress with systematic planning that informs institutional managerial and financial decision making.

A common theme apparent throughout the site visitor's report, the Chancellor's responses, and the October 19, 2000, CIHE findings is that during the decade of the 90's our mission and range of activities have expanded rapidly, more rapidly than the resources available to manage change. Both the visiting team and the CIHE agreed with our own statement in our self-study that "we remain in the earlier stages of developing integrated planning and evaluation."

During the '90s a great deal was accomplished. The campus grew in size, faculty work was recognized with awards and accolades across many disciplines, national and international accreditations were achieved or maintained in the professional schools, both graduate education and funded research activity expanded dramatically, and US News and World Report ranked the campus in the Top Ten Regional Publics in the Northeast. There was increased campus outreach and connection with the external community, which provided both public recognition for achievement and opportunities for further growth. While we embrace these developments as important for the campus we agree that few were implemented as the result of systematic planning. Although a variety of more or less formal planning activities occurred at UMass Dartmouth - summarized in detail in our NEASC 2000 Self Study - some actions were not done with direct reference to how they furthered larger goals and some were not linked in a transparent way to the annual or long-term budget planning processes. Internal campus budget allocations were unable to keep pace with the level of activity initiated. Some units felt that their issues did not get the attention they deserved. In sum, during this period of growth and change balancing planned vs. entrepreneurial activity was a constant challenge.

3.1.1 Context for our present planning effort

The challenge of the planning efforts of the '90s is perhaps rooted in the history of the University of Massachusetts Dartmouth - evolving from a number of technical institutes and institutional mergers into a university offering a broad range of liberal and professional education. This occurred, for the most part, without a comprehensive planning process. Opportunities presented themselves, and when there was agreement a synergy of aspiration, change, and growth happened in the profile of students, faculty, and programs. At each stage of institutional development mission and vision statements served as only a broad guide for future development, and detailed planning strategies and budgets were not always prepared.

In 1991, when the campus became part of University of Massachusetts system, the Board of Trustees of the University of Massachusetts, now a five-campus system, adopted a vision statement for the whole university and encouraged each campus to develop its own distinctive role within the university. Organized planning was encouraged and supported. UMass Dartmouth was identified as providing excellence in undergraduate education, becoming a high quality regional university, and urged to continue its development of selected graduate, research, and outreach programs. These system statements provided broad direction for efforts over the last decade and allowed us to achieve national accreditation for the College of Business and to expand programs in engineering and marine science.

In 1992 an Academic Planning Task Force was created to make these broad goals more concrete. The Task Force first prepared a one-page UMass Dartmouth Vision Statement, which was adopted in that first year. This group then developed a planning document called "Building On Our Strengths," which emphasized the beneficial relationship between undergraduate and graduate education, strengthening commitment to teaching and public service, expanding research activity, and supporting programs to serve the educational needs of the Commonwealth's increasingly diverse population. This plan was updated in 1995 as "Building on Our Strengths Revisited." It, in turn, led to a document generated by the faculty, the Shared Academic Agenda (Spring 1997), to which was added a Graduate Planning Statement (Spring 1998); both were approved by the Faculty Senate and adopted by the administration. The Shared Academic Agenda made the claim of being "the basic planning document of the university" (President of the Faculty Senate, memo conveying the final document to faculty, librarians, and professional staff, March 12, 1997). The campus considers these as foundation documents.

3.1.2 Recent planning activity

In the fall of 1999, Chancellor MacCormack and Provost Curry initiated a campus-wide strategic planning activity. In this effort our goal was to build on past planning efforts, broaden campus and off-campus constituent involvement in the process, and lead the campus to its own next level of aspiration as an established member of the UMass System. This new administrative leadership made a strong commitment to planning not only as a tool to develop strategies for the institution's future but also as a process to manage change systematically at the various institutional levels. They made a deliberate effort to establish processes that would link planning goals to resource allocation and clearly correlate intentions and outcomes. Figure 3.1 illustrates the planning activity initiated during 2000-2001.

A consultant was retained to help guide the gathering and assimilating of data, and the Chancellor appointed a 28-member ad hoc planning team. The team used brainstorming methodologies to decouple visionary thinking from the encumbrances of the present situation. (Documentation of this process, including a list of team members, is provided in the Team Workroom.) Nearly 1,000 requests were made to individuals on and off campus to assess UMass Dartmouth's strengths, weaknesses, opportunities, and threats (SWOT's). Each team member wrote a "homework" essay on an assigned central topic. The purpose of these activities was to gather as much data as possible so all concerned would have a common understanding regarding important campus issues. These essays, along with the SWOT data, previous planning documents, and the NEASC 2000 Self-Study Report, formed the backdrop for a three day off-campus meeting during which the team developed a draft campus mission and ten-year vision and engaged in an exercise to compose strategies and tactics for implementing the vision.

Figure 3.1: History of Planning Activity

To communicate this work to the campus community, the Chancellor and the Ad-Hoc Planning Group held an open forum in the spring of 2000. During the remainder of the spring 2000 semester, various smaller groups (college and departmental units) on campus met with assigned members of the ad hoc planning team to discuss the draft mission and vision. Feedback from these discussions led to some modifications of the draft mission and vision and indicated clearly that many members of the campus community, while they generally supported the vision and mission, were anxious to develop action strategies to advance the planning framework. During the summer, sub-committees continued to meet to devise communication strategies for dissemination of information, develop a narrative description of the effort, present sample data and benchmarks, and incorporate feedback into the draft mission and vision statements.

In September 2000, the planning team met to review the status of the campus-wide effort and plan for the Chancellor's open forum held on September 27, 2000. At this forum there was a clear sense that the mission and vision statements resonated well with most campus constituencies. However, discussion continued about translating plans into specific goals, priorities, and pathways of implementation and linking the results to available fiscal resources. The specifics of the growth strategy were the subject of broad discussion.

Consideration of the results of this campus feedback led to a decision to encompass the vision and draft plans within nine priority areas, each represented in a focus group. The Ad Hoc Planning Group completed its formal work as of the date of this forum. Volunteers were recruited from faculty and staff to be on the nine focus groups. Several members of the Ad Hoc Group continued to participate as members of focus groups.

The nine focus groups each had a faculty member and staff member as co-chairs, and close to 150 faculty and staff agreed to participate in the groups. The Chancellor, the Presidents of the Faculty Senate, the Student Senate, and the Faculty Federation, and the Vice Chancellors for Academic and Student Affairs assigned the various committee members to their first or second choice groups and agreed on the initial co-chairs. The co-chairs formed an ongoing University Planning Council (UPC) to guide the next phases of the planning effort and produce a written plan, gather appropriate input from various constituencies, and recommend it for action to the Chancellor. Figure 3.2 summarizes the organization of this planning effort. The first task of each focus group was to flesh out the initial strategies; see if they were clear and complete; and add, subtract, and "focus" on a practical agenda in each area. Each of the groups formulated a work process under established timelines. The broad definition of each focus area is summarized below:

Figure 3.2: Organization of the Planning Effort

• Develop and enhance academic programs. Improve and sustain quality of existing academic programs and create plans for new undergraduate degree programs, expanded graduate programs, the integration of theory and practice, and the application of experiential learning activities into all majors.
• Develop and sustain a service culture. Improve and streamline work processes and foster responsive services and service delivery in all areas including HR, AA/EEO, facilities, CITS, campus services, student services, academic support, and fiscal services.
• Create campus spirit and build community. Build the sense of community and school spirit on campus, including an increased integration of curricular and co-curricular activities to enhance the holistic development of students.
• Stabilize fiscal resources and link them to goals. Stabilize the UMass Dartmouth fiscal situation by expanding revenue and managing resources well; ensure that there is a functional, clear, consistent budget process, along with improved fiscal management tools and processes; communicate clearly how fiscal resources are linked to goal activity.
• Develop a facilities master plan to guide facilities and technology improvements. Develop a comprehensive facilities master plan, including a technology infrastructure plan that includes a facilities inventory, interim and long-term space planning strategy to guide facilities and technology improvement plans in all areas of campus facilities.
• Invest in faculty/staff development. Increase ongoing support for professional development activities, including assessing opportunities for faculty/staff development.
• Make connections and form partnerships. Become truly embedded in our community by broadly engaging the intellectual capital of our faculty, staff, and students through partnerships and outreach.
• Manage enrollment. Improve campus enrollment planning and implementation through improved communications among chairs, deans, registrar, advising center, the fiscal area, and housing regarding enrollment decisions and by forecasting accurate projections of student targets and management of enrollment growth.
• Develop meaningful outcomes assessment. Improve institutional effectiveness and student learning through on-going and diversified assessment.

During the spring 2001 semester, these focus groups did voluminous work on strategies for improvement in each focus area and held meetings to generate a large number of ideas about what was important to accomplish. Approximately 100 members of the UMass Dartmouth community were focus group participants.

In parallel with continuing efforts to write a comprehensive plan narrative from this work, the Chancellor decided to apply evolving plan concepts in a number of specific demonstration projects in the 2001-2002 academic year. These demonstration projects were intended to lead to practical results, with the goal of increasing confidence that a planning process could lead the campus forward, and to help create consensus on the parameters for a final comprehensive plan.

3.1.3 Current directions and achievements

The mission and vision statements (See Appendix 2, items 9.2 and 9.3, for the current draft versions) continue to guide the work of the UPC. During the summer of 2002 some members of the UPC and the focus groups left the University; some took advantage of the State of Massachusetts early retirement plan, while others moved on to positions in other institutions. The remaining members of the UPC re-committed themselves to the development of a written plan. The fall of 2002 was dedicated to reconstituting the focus groups, with many new members of the University community joining this effort.

In two important instances, work done by the focus groups in 2001-2002 has been implemented.

• Based on the successes of its demonstration projects during 2001-2002, one of the focus groups, Enrollment Management and Planning, has been transformed into a permanent and integral campus activity, coordinated by the Provost's office. Members of the previous focus group act as an advisory panel for enrollment planning. Representatives from Academic Affairs, Student Affairs, and Fiscal Affairs - including the Directors of Housing, Admissions, and Financial Aid, and the Registrar, Special Assistant to the Provost, Associate Vice Chancellor for Academic Affairs/Graduate Studies, Associate Vice Chancellor for Fiscal Affairs, Associate Vice Chancellor for Continuing Education, and Associate Dean of the College of Arts and Sciences - and faculty (the members of the Faculty Admissions Committee) meet as a team to recommend on the range of enrollment management issues and propose short-range and long-range enrollment goals. In mid-fall 2002 this group developed a formal projection of enrollment growth to 2010, which was reviewed by the UPC and presented to the Chancellor for approval. Projecting levels and types of students (undergraduate, graduate, continuing education; new, readmitted, and continuing; students living on campus) each year through 2010, this work will be used in budget and facilities planning as well as other planning activities, as reported in later chapters of this report. The 10-year enrollment projection is included in Appendix 2 as item 9.5. This projection will be updated yearly and is currently being expressed at the program level.
• A second focus group activity, the development of a facilities master plan, is also becoming institutionalized, as reported below in the chapter on Focus Area C. The consultant and steering committee appointed by the Chancellor will meet with the UPC Facilities Planning Focus Group to review their work's consistency with overarching campus priorities.

These two results serve as models for the future functioning of the UPC. As plans in each group mature, the group will refer the plans to the appropriate administrative individuals or bodies for implementation. The UPC members will continue to lead their focus groups in additional planning activities and will inform the UPC of these activities.

On February 3, 2003, the draft of the mission and vision statements were presented to the Faculty Senate for approval. Discussion brought out the need for a special meeting of the Senate, which will occur February 18. In addition, a draft statement of goals and objectives was presented for review and input. In spring 2003 the UPC will finalize the goals and objectives of the plan and present the final plan to the Chancellor. The Chancellor will consult with the Faculty Senate and others on campus before giving the plan her final approval. This process should be complete by June 2003. The NEASC visitors will be appraised of its status during their visit.

3.1.4 Systematic planning in the divisions and colleges

Student Affairs, Academic Affairs, and the Colleges have formal planning processes. Each has adopted (or is in process of producing) a written statement of mission, goals and objectives, and specific actions or initiatives. Copies of background documents are placed in the Team Workroom.

• The College of Arts and Sciences created a college plan during 2001-2002 which is now used by the Dean to guide allocation of resources and provide a conceptual structure for curricular program change, as discussed below (see page 26).
• The College of Engineering uses comprehensive planning to guide resource allocation, curriculum change, and development of new programs. Its strategic plan was created in 1999 and is now under revision in recognition of recent changes in resource availability.
• The College of Nursing developed a formal, written comprehensive plan in 1995. All of the goals of that plan were met by 2000. The faculty then undertook revision of the undergraduate curriculum, based on evaluative data from students, alumni, and changes in the profession. That work is in its final stages of completion and approval and will be forwarded to the Massachusetts Board of Nursing and the National League for Nursing Accrediting Commission for their approval. The new curriculum will be implemented in January, 2004. In the Spring of 2003 the College of Nursing will commence a new strategic planning activity. This plan will utilize the university's goals and objectives to determine the College's response and action to further these goals.
• In 1999-2000 the Charlton College of Business formulated a ten-year plan for 2000-2010 as a required element in AACSB accreditation. Briefly, the plan encompasses expanded enrollments, program and faculty development, fiscal and administrative support, building connections to employers, and program image and recognition. To date, the plan has guided efforts to enhance case-study methods across the curriculum, expand instructional technology, improve student membership in professional organizations, and plan an Executive MBA version of the present MBA program.
• The College of Visual and Performing Arts is in the process of completing a strategic plan. Mission statements have been adopted for each department and the college as a whole. In addition, departments are developing long and short term goals and plans for their implementation. Goals for the college include expanding interdisciplinary programs; reaching out to the university community through more general education course offerings; improving the academic qualifications and cultural diversity of incoming students and having more be international; ensuring more rigorous assessment of students' progression to the second and third year of studies; developing curricula to better reflect diversity and globalization; strengthening relationships with alumni, in part to enhance student placements; and continuing to develop outreach, both K-12 and for community cultural enrichment.
• The Division of Student Affairs created a strategic plan in 1998-1999. Based on a mission statement and analysis of strategic challenges and initiatives, the plan presents thirty-one specific strategic initiatives. A careful review in Spring 2002 of the results of these initiatives showed that particular areas needed additional attention. The following Student Affairs Advisory Groups were created to address the targeted areas of Campus Community, Campus Climate, Campus Policies and Procedures, Communication, and Campus Programming. These groups meet regularly, break down into work groups, and make recommendations to the Vice-Chancellor for Student Affairs.
• The Provost initiated an action planning process for the Division of Academic Affairs in fall of 2002. The plan describes a mission and strategic priorities for the division; presents three main goals (enhance the teaching and learning environment, expand scholarly/creative activities including externally-funded endeavors, and regularize resources and demonstrate academic leadership); and defines objectives to accomplish the agenda. Each objective is supported by a range of specific action items - from "Enhance student-centered learning in curriculum" to "Establish a network/support group among seasoned, established researchers/scholars/artists to mentor colleagues in the colleges and disciplines" - that are assigned to members of the Provost's Council with specific timelines for completion. Action plan progress is a standing agenda item for the Provost's Council.

College and Division planning documents will be up-dated and re-focused for alignment with the campus plan after it is final this spring.

3.1.5 Next steps in institutional planning

From the beginning of UMass Dartmouth's current planning effort it has been clear that there is a mandate from the University community not to allow this plan to "sit on the shelf." An explicit goal within the strategic plan (goal 6) calls for the UPC to remain an on-going committee of staff, faculty, and administrators which will review planning progress and implementation in each goal area, develop subsequent priorities and goals, and tie new goals to budget planning and allocation. The UPC will meet at least three times a year. At the beginning of the academic year the committee will determine key projects that advance the goals and tie appropriate resources to those projects; they will meet again in mid-year to review progress; and at the end of the academic year they will meet to determine progress and plan for the following year. Once the plan is approved by the Chancellor, it will be utilized by each division in the University to focus both long-term and short-term goals and activities, as appropriate for that unit.

To summarize our status in implementing systematic planning, the University of Massachusetts Dartmouth has made substantial progress toward being an institution with an institutional plan that directs growth and development and links planning and resource allocation. Data driven decisions will tie goals to resources; and evaluation and review, at multiple levels, will become the way the university measures its successes and guides future changes. The full, written plan will soon be made final, approved by the Chancellor, and presented for implementation, so that planning can become an on-going campus process referred to continuously in decision-making. Finally, evaluation feedback needs to be introduced into the planning process to measure successes, inform next steps, and indicate necessary adjustments.

3.2 Assessing student learning outcomes

The NEASC 2000 visiting team noted that the University's efforts in student learning assessment were "in a relatively embryonic form and being used only to a limited degree." This section examines progress in assessing academic major programs; assessing the general education program; and assessing student engagement and a new project with learning communities.

This section will show that UMass Dartmouth's progress in doing assessment reflects a commitment to student success that has at its core a commitment to examining whether students learn what we think they should know by the time they graduate.

3.2.1 Assessment in academic programs

All the majors in Business and Nursing and most majors in Engineering and Visual and Performing Arts must satisfy external accreditation requirements that include assessment. In addition to that required by external accrediting entities, UMass Dartmouth has combined assessment with a periodic planning and evaluation review of every academic program, called the Academic Quality Assessment and Development or AQAD review. The Commission on Institutions of Higher Education specifically requested information on progress with AQAD reviews in its October 19, 2000, report of findings. AQAD reviews require external expert reviewers and result in an action plan approved for each department by the dean of the college. Programs subject to external accreditation reviews substitute those reviews for the input portions of the AQAD review, but they also produce AQAD action plans for implementation. Appendix 2, item 9.7, contains criteria, procedures, and a timetable for AQAD reviews. At the conclusion of the present academic year UMass Dartmouth will have completed one cycle of AQAD reviews.

The following paragraphs describe accomplishments and challenges in each college.

Arts and Sciences. The College of Arts and Sciences is UMass Dartmouth's largest college. It serves almost half the undergraduate majors at the university, delivers most of the general education and honors instruction, and has the largest number of faculty. Of the five principal goals in the college's strategic plan (available in the Team Workroom), the first two give undergraduate and graduate instruction the highest priority. Appropriately, the plan also commits the college to assessment as a way to "improve our curricula and how we deliver them." The plan gives this commitment "teeth" by noting that the college will allocate support to college programs based on certain priorities, including: "quality of instruction, as demonstrated in materials associated with annual performance reviews, student evaluations, tenure files, promotion files, and learning outcomes assessments to improve curricula and instruction."

The College of Arts and Sciences has made considerable progress toward realizing its commitment to assessment. The major indicator of this progress is that 12 of 14 departments in the college have identified what they expect their undergraduate and graduate majors to know by the time they get their degrees. The remaining two programs (Biology and Chemistry) will have done so by the time they complete their AQAD reviews in 2003. These departments have graduate programs and will develop assessment plans for them too; the other two departments with graduate programs, English and Psychology, have assessment plans for them.

Copies of the college's undergraduate and graduate major assessment plans from the 12 departments are provided in the Team Workroom. While these materials vary in their structure and content, taken together, they demonstrate an understanding of assessment and a willingness to do it. The following list summarizes some of these department plans.

• One of the most complete sets of materials is from the English Department. English sorts its assessment materials by undergraduate (literature, drama and film, and writing and communications) and graduate (professional writing) concentration. These assessments include multiple sources of assessment data and data collection strategies and, most importantly, a commitment to write an annual report "to the department identifying areas of strength and weakness...and appropriate adjustments addressing weaknesses and reinforcing strengths in the curriculum." The external reviewers for the English Department's AQAD review found the department very advanced in its design of an outcome-based curriculum.
• Economics has chosen a course-embedded strategy to do assessment. After identifying what faculty think students should know by the time they graduate, the Economics assessment plan includes a matrix that ties each learning goal to a particular course or set of courses. The faculty use materials from these courses to build a portfolio of work completed by each student and a senior year research paper to assess what those about to graduate have learned. The plan rightly notes that the faculty expect their assessment process to evolve over time as they annually consider the effectiveness of their plan and curriculum.
• The Portuguese Department was established last year and used its AQAD review to begin developing its assessment plan. It has made an excellent start in identifying learning goals (e.g., oral fluency and familiarity with terminology and methods of critical analysis). While the faculty's current approach to doing assessment focuses on using class-based performance indicators, the faculty are appropriately "cautious" about "placing an excessive weight on these imprecise instruments."
• Political Science has the least developed sense of what it should do. Its faculty are considering using samples of student work collected over time to measure the "added value" contributed by coursework over time. While this is an interesting strategy, the department needs to develop a better and more specific vision of what is being measured than "understanding of content in political science."

As these four examples show, different departments in the College of Arts and Sciences are at different stages of development in creating their plans for assessment and are using different approaches. The Dean is conducting a plenary faculty meeting this spring at which he will stress moving from creation of assessment plans to generating assessment results; he will use the results in future decisions about allocation of resources, including faculty hiring.

Finally, the college is requesting that department assessment materials, especially information on learning goals, be shown in the General Catalogue to give greater visibility to curriculum design, expose students to learning goals, and encourage accountability.

Business. Each department within the Charlton College of Business continually monitors and assesses its curriculum. The process relies on input from faculty, students, and outside stakeholders. These efforts are tied closely to criteria established to maintain accreditation by AACSB International.

Every year each department surveys its graduating students. The surveys solicit feedback regarding the college curriculum and curricula within the majors. Students are also asked what each department is doing well and what areas could use improvement. Outside stakeholders are represented on the various Departmental Advisory Committees, whose members are regional employers and business college alumni. The Committees meet at least once a semester and provide feedback on the current curriculum and make suggestions for future improvements, based on their perspectives as former students now active in the business world and as current employers of UMass Dartmouth graduates. One outcome of external feedback and assessment has occurred in the Accounting major. Based on feedback from its advisory group, the Accounting faculty has proposed a new Accounting Information Systems option. The Departmental Curriculum Committee approved the option during the fall 2002 semester and has passed the proposal up to the College Curriculum Committee for action.

In accordance with such feedback, reviews of best practices, and criteria for AACSB accreditation, faculty consistently monitor the curriculum. Courses (syllabi, readings, assignments and projects, papers, and exams) are periodically reviewed to ensure that subject matter and AACSB competency areas are covered.

Following are some recent results of these activities:

• The first-year course GBA 101, originally a one-semester general introduction to business course offered by one department, is now a two-semester, college-owned interdisciplinary course introducing students to the AACSB competency areas through an internet based module called Biz-Tech. Biz-Tech provides an overview not only of business but its external environment and issues regarding ethics and social responsibility. Each student creates a business plan which ties in all of the major business area. GBA 101 also provides the basis for first and second year student advising.
• Assessment of outcomes led to an upgrading of admissions standards and revision of the core statistics requirement. With the implementation of new admission standards, the average Math SAT score increased 24 points from 509 in 1995 to 533 in 2000. As a result, the need for a "remedial" approach in the statistics course has been reduced, and the course is being changed from a two semester sequence to a single semester course concentrating on applications to business. Each department has been charged with integrating specific statistical skills and tools in upper division courses in each major.

Programs in business were last evaluated in 1999-2000 by AACSB International in the review that resulted in accreditation by that body. This activity also encompassed the AQAD action planning process. The Charlton College of Business's evaluation program is designed to meet the requirements for maintaining AACSB International accreditation, which requires that "each degree program should be systematically monitored to assess its effectiveness." At this point, the college's approach is based on evaluating accomplishment of mission and accreditation standards; accomplishment of learning objectives is a secondary consideration.

Engineering. The following undergraduate programs in the College of Engineering are fully accredited by the Accreditation Board for Engineering and Technology (ABET): Civil and Environmental Engineering, Computer Engineering, Computer Science, Electrical Engineering, and Mechanical Engineering. Results of the ABET reviews of these programs are incorporated in the required AQAD Action Plan. At mid-cycle between ABET reviews, these units conduct an AQAD review of their graduate programs. Physics and Textile Science follow the AQAD review criteria, and by college policy do so in parallel with criteria and processes used in ABET.

ABET established a strict system of outcomes-based assessment with its Engineering Criteria 2000. By these criteria,

Engineering programs must demonstrate that their students have (a) an ability to apply knowledge of mathematics, science, and engineering; (b) an ability to design and conduct experiments, as well as analyze and interpret data; (c) an ability to design a system, component, or process to meet desired needs; (d) an ability to function on multidisciplinary teams; (e) an ability to identify, formulate, and solve engineering problems; (f) an understanding of professional and ethical responsibility; (g) an ability to communicate effectively; (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context; (i) a recognition of the need for, and ability to, engage in life-long learning; (j) a knowledge of contemporary issues; and (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

The college has been preparing for outcomes assessment for several years, anticipating the accreditation visit scheduled for fall 2004.

All undergraduate programs in the college have established statements of goals and objectives that appear in the General Catalogue. The process that led to these statements was fully interactive. Each department developed a draft of its goals and objectives and vetted them with their Industrial Advisory Boards before finalizing the language. Some programs have gone through at least two iterations to arrive at the current statements. Department faculty then worked with each of their courses and produced specific outcomes carefully articulated with both departmental and ABET objectives. Again, an interactive process was adopted. Some departments have progressed to the stage of modifying their original statements based on feedback from their constituencies, including alumni.

As a further step in implementing assessment, three programs (Computer Engineering, Electrical Engineering, and Mechanical Engineering) have begun to assess student performance using embedded outcome-coded questions on examinations, term papers, and class projects. Average student performance is being calculated and reported for each question on each examination in fall 2002; this process will be routine for future course offerings. The college expects to have similar mechanisms in place and operating for all of its programs by fall 2003. Programs that have already instituted the outcome-coded assessment process will close the feedback loop during the 2003-04 academic year and make appropriate changes to improve the delivery of their course contents.

The College of Engineering applies focused assessment of student outcomes in its IMPULSE program, an innovative program that presents an integrated first-year curriculum to calculus-ready first year students, modeled after research-tested engineering curricula at the NSF-funded Foundation Coalition Universities. In addition to specific curricular goals, the program includes such primary goals for students as "become motivated life long learners" and "work effectively in teams." Results have been assessed using a variety of both qualitative and quantitative approaches under guidance from an IMPULSE Assessment Committee, comprising faculty from each department involved with IMPULSE. Performance measures include test scores, grades, credits earned, and retention/graduation rates, and these are reviewed against nationally normed comparisons. Every year IMPULSE students complete surveys about their attitudes and experiences, for example, how well they feel they have progressed toward ABET-specified learning outcomes like teaming and lifelong learning. The IMPULSE survey process has been expanded to a college-wide, web-based survey system, comparing results for students in different College of Engineering majors, in different years of study, and in or not in IMPULSE. The IMPULSE Assessment Committee is also conducting a program review of IMPULSE this year.

As mentioned previously, graduate programs in Computer Engineering, Electrical Engineering, and Mechanical Engineering are assessed in AQAD rather than ABET reviews; the reviews occurred in 2001-2002, each producing an action plan for improvement. The departments and Dean have found the process valuable and a very appropriate supplement to the review that they do of undergraduate programs for ABET. The Computer Science graduate program will join that group in the next AQAD cycle. Action plans address such matters as, for Computer Engineering and Electrical Engineering, continuing to develop research orientation in the faculty, focusing graduate student admissions better to match faculty depth and expertise, and pursuing curricular cooperation with Computer Science; and for Mechanical Engineering, increasing the number of qualified graduate students, changing curriculum pathways to give students greater flexibility of options for specialization, and enhancing graduate student advising.

Nursing. A program evaluation plan was developed by the College of Nursing Evaluation Committee and approved by the faculty in 1996. The plan provides a systematic process for gathering data to develop, maintain, and revise components of the instructional program. The plan identifies a time frame for evaluation, methods of evaluation, parties responsible for gathering data, methods or instruments to be used, and the feedback communication loop. The Evaluation Committee monitors implementation, makes regular reports, and tracks data collection. The plan, itself, is evaluated and updated by the Evaluation Committee. The current College of Nursing evaluation plan is included in the NEASC team workroom.

The plan focuses on assessment of learning outcomes through a systematic process to evaluate student performance relative to critical thinking, communication, and therapeutic interventions. Believing that nurses must be able to draw reasonable conclusions about the clients for whom they provide care, the faculty decided to administer to students in the undergraduate and graduate programs in nursing a standardized test sequence that measures critical thinking, the California Critical Thinking Disposition Inventory and the California Thinking Skills Test. Students were tested upon entry into the program, mid-point in the program, and upon graduation. Consistently low mean scores on the inference sub-scale for the classes of 1997, 1998, and 1999 were of particular concern, and several strategies were identified to build inference skills in students. At the University level, the General Education requirements implemented in 1998 may assist in the development and refinement of those variables, particularly in reference to a new ethics component. At the College level a "white paper" highlighting teaching strategies for inferential thinking was developed and distributed to faculty, and the Center for Teaching and Learning also offered Nursing faculty a strategy-teaching workshop. Preliminary review of data for the classes of 2000, 2001, and 2002 indicates that many faculty utilized these new teaching strategies, and a gain in scores from a mean of 4.33 to a mean of 4.89 is noted.

Visual and Performing Arts. During the past two years, the departments in the College of Visual and Performing Arts (CVPA) have been developing learning outcomes and assessment plans in preparation for the upcoming reaccredidation visit by the National Associations of Schools of Art and Design (NASAD), which occurs Spring 2003. At the same time, the Music department will be undergoing an AQAD program review. This work responds to new NASAD requirements for assessment of learning outcomes as well as UMass Dartmouth priorities. The finished documents will be available in the Team Workroom.

CVPA has a long-standing and systematic practice of assessing learning in the form of critiques ("crits"), juries, and recitals that monitor students' progress as artists. Critiques and juries have always been important midterm and semester-end activities in the visual arts, allowing the professor and fellow students to examine and discuss each student's progress and serve as review steps for progression to advanced program levels. Reviews at the end of the freshman year help place beginning students into the most appropriate studios. Then, at the close of the junior and senior years in the visual arts, students exhibit their works publicly. Art Education students do a senior project and present results in a juried setting. In the performing arts, both Music and Theater classes use juries and presentations, and Music students give junior and senior recitals. Faculty have also developed assessment and outcome criteria for historical and theoretical courses, tying their outcomes to those of applied and studio courses. MFA students prepare both visual and written theses, which they present in juried settings.

SMAST. The School for Marine Science and Technology faculty have developed learning outcomes and measures as they begin to do assessment; the program is just initiating assessment with a first class of students entering this fall. The SMAST assessment plan, available in the Team Workroom, was developed after a review of similar documents used by science departments at leading institutions and was outlined in the proposal that led to approval of their academic program.

Support for assessment activities in the colleges. Additional evidence of the University's commitment to assessment is its support of an initiative to design and test the use of electronic portfolios. Two faculty members have developed a Learning Portal to support assessment-based continuous improvement in both traditional and team-based classrooms. Faculty can post both assignments and assessments (e.g., surveys, quizzes, etc.) to the Portal, which stores student assignments and their assessments. Faculty can use such student submissions to continuously improve the course while they engage the students in learning by having them evaluate their coursework throughout the semester. The project also offers a set of assessment tools to allow faculty to develop surveys and other assessments and to generate reports automatically. Faculty are testing these tools in their classes this spring, and the Provost has provided funding for staff support to maintain and develop the Learning Portal server and the associated assessment tools.

As our colleges, academic departments, and other units become increasingly involved in assessment activities, there is increasing need for technical and logistical support. Units seek assistance in such immediate matters as designing and administering surveys, tabulating survey results, analyzing quantitative and qualitative findings, and keeping systematic, on-going records; and in more substantive aspects of approach and design for assessment programs. Such resources are limited at this time. The Center for Teaching and Learning, founded in 1998, has become a source of assistance in aspects related to teaching methodology and faculty development. A part-time assessment specialist in the Office of Institutional Research focuses primarily on assessment of the IMPULSE program (discussed on p. 30 above) but provides some technical assistance to other units and also serves as a member of the UPC Assessment Focus Group. Planning is underway to expand technical support services for assessment.

3.2.2 General education assessment

UMass Dartmouth's general education program was approved by the Faculty Senate and the campus administration in spring 1997; a revised and enhanced program was approved the following fall; a General Education Committee began reviewing course proposals for inclusion in the program in early 1998; and the curriculum went into effect for the Fall 1998 entering first year class.

The goals of the general education program are to educate students to be proficient in areas independent of their majors and to (a) have a working understanding of the connections between disciplines; (b) appreciate and respect the differences among us; (c) be ethical, culturally-aware, and socially responsible; (d) be quantitative, rational thinkers; and (e) be effective and creative communicators.

UMass Dartmouth's general education program offers long lists of courses in each of the following categories: Cultural and Artistic Literacy; Ethics and Social Responsibility; Mathematics, Natural Science, and Technology; Global Awareness and Diversity; Information Technology and Computer Literacy; and Written and Oral Communication. Departments either develop new courses or adapt existing courses to meet stated criteria that can qualify them for inclusion in one of the general education categories. This design has given many departments an incentive to participate in the program and also gives students maximum flexibility in choosing courses. But it has also complicated the assessment of the program's effectiveness by requiring assessment of hundreds of courses in a wide range of subject disciplines. Assessment must explore success in realizing the stated criteria within each course.

Assessment activities began in Fall 1999, with the formation of project teams of faculty members, program directors, and staff. These teams developed outcomes for four content areas: Cultural and Artistic Literacy; Global Awareness and Diversity; Ethics and Social Responsibility; and Math, Natural Science, and Technology.

In spring 2000, the Faculty Senate approved the General Education Committee's recommendation for an Assessment Task Force. It was formed the following fall and considered several approaches to assessment, using outcomes developed by the project teams; it presented its recommendations to the Faculty Senate in April 2001. The Senate voted at the April meeting to develop and implement a pilot assessment plan in the area of Mathematics, Natural Science, and Technology.

The Provost appointed a committee to coordinate and advocate for assessment in the summer of 2001. It included the Director of General Education, the chairs of the General Education Committee and the Faculty Senate's General Education Assessment Task Force, and the Dean of the College of Arts and Sciences. The committee's first task was to act on the Senate's vote to develop and implement a pilot assessment plan in Mathematics, Natural Science, and Technology in the 2001-02 academic year. The committee also initiated assessment in the Writing and Critical Thinking (Tier II) skill area. The committee initiated assessment activities in two additional areas (Global Awareness and Ethics and Social Responsibility) in 2002-03.

The committee's approach to assessment is to form two new faculty groups at the beginning of each academic year and to ask each group to develop a pilot assessment plan (including outcomes, measures, and data collection strategies) in a particular content or skill area by the end of the fall. Each group then implements its plan in the spring and uses its evaluation of the experience to refine its plan for use the following year. Materials in the Team Workroom include agendas from breakfast meetings in September 2001 and 2002 when each of two pairs of faculty groups was formed, lists of faculty who volunteered to work on each group, assessment tools, and evaluation reports from the first pair of groups.

The following paragraphs review activities from 2001-02 and 2002-03. Complete documentation is available in the Team Workroom.

Tier I Writing. Faculty responsible for English 101 (Critical Writing and Reading I) and English 102 (Critical Writing and Reading II) have primary responsibility in this skill area. Students must take the Freshman Writing Placement Test before they can enroll in English 101. Unsatisfactory performance on the test requires students to take special sections of English 101 that require skill-building laboratories in addition to regular course requirements.

Tier 2 Writing and Critical Thinking. The Board of Higher Education's Assessment of Academic Proficiency Task Force distributed a draft paper in spring 2001 describing proposed outcome proficiencies in writing and critical thinking and some recommended assessment instruments. (The Board soon dropped its proposals and asked each UMass campus to use a campus-specific approach.) To respond to the Board's mandate, in June 2001 UMass Dartmouth invited representatives from American College Testing (ACT), Education Testing Services (ETS), UMass Boston, and UMass Dartmouth's College of Nursing to discuss assessment with interested faculty and administrators, and that summer the Provost targeted writing and critical thinking as the first assessment area to initiate. The committee that formed in September adopted a course-embedded approach, identified appropriate outcomes, and designed a matrix that faculty in any discipline can use to assess written assignments. This matrix is shown in Appendix 2 as item 9.6. The matrix was piloted in spring 2002 and is being refined and revised in the current academic year; a report on its use was given to the Faculty Senate in fall 2002. Expanded faculty use of the matrix is being emphasized in 2002-03.

Mathematics, Science and Technology. The Provost's committee also formed a faculty group in fall 2001 to honor the Faculty Senate's vote to develop and pilot an assessment plan in this area. The group adopted a course-embedded approach; they developed different assessment methods to pilot in different courses in Biology, Chemistry, Mathematics, Medical Laboratory Science, and Physics. These methods, including pre- and post-tests and portfolios, were piloted in spring 2002 and results were reported in fall 2002 to the Faculty Senate. The result is a selection of course-embedded assessment designs for science departments to adopt.

Global Awareness and Ethics and Social Responsibility. The Provost's committee formed two additional groups in fall 2002 to develop outcomes and pilot assessment plans for these areas. This last fall these groups have produced pilot assessment plans that will be initiated in Spring 2003.

Contribution of the campus library. The campus library contributes to the information literacy component of the general education program. Instruction is done through partnerships between content faculty and library faculty in English 101 and 102 classes. The library has two years of survey data from students and instructors on this initiative. The effects of it on learning are examined in a report written by library faculty (in the team workroom).

To summarize the status of general education learning outcomes assessment, much has been accomplished but work remains to be done to move from piloting to broader implementation. The Provost's Committee on General Education Assessment will form additional groups next fall and in successive fall semesters to develop assessment plans in the remaining content/skills areas. The committee will continue to work with the first four groups on refining learning outcomes, measures, and other elements of their assessment plans and on expanding the number of faculty participants. The intention of the Provost's committee is to create an information base describing whether student learning and performance indicate that the general education program is successful or whether its curriculum and the way it is delivered require refinement or substantial change.

3.2.3 Student engagement and learning communities

A current initiative related to developing learning communities at UMass Dartmouth - programs that foster student-centered learning by grouping students, linking curricula, and integrating curricular and co-curricular experiences - offers an opportunity to do assessment that examines the effect of the quality of campus life variables on learning. The Provost led a team of UMass Dartmouth faculty, administrators, student affairs professionals, and a student to a learning communities workshop at Evergreen State College in June 2002.

UMass Dartmouth had experience related to learning communities before the trip. The Engineering College has a learning community initiative called the IMPULSE Program (discussed above, p. 30). And the UMass Dartmouth Advising Center has enrolled first year students in "packets" for some time; the packets have the characteristic of learning communities in that students in each packet take several courses together. While the Center did nothing beyond creating these groupings, they gave a starting point for creating true learning communities. An assessment plan is integral to the project, part of which examines whether learning communities that involve co-curricular activities and residence hall activities and staff influence learning. The plan includes the use of survey data on student behavior and attitudes describing the quality of intellectual life on campus, study habits, and student-faculty relations.

The survey referenced above is itself based on some topics and questions asked in the National Survey of Student Engagement (NSSE). UMass Dartmouth has participated in this survey in spring 2000 and spring 2002, along with other University of Massachusetts campuses. The NSSE survey provides an important source of information about undergraduate students across the institution. Asking freshmen and seniors questions about Academic Challenge, Interactions with Faculty, Enriching Educational Experiences, and Supportive Campus Environment, the survey gives information about such issues as the time students spend on homework, the papers they write, their extra-curricular engagement, and the academic enrichment activities that we require or they seek out. Members of the Provost's staff are currently analyzing the results of the spring 2002 administration. Our NSSE rankings within our national comparison group, the Masters institutions, indicate that there is need for improvement. On the other hand, we believe that students' responses to specific questions in the various NSSE categories reveal more that is useful in designing interventions than do the overall peer-group rankings. After the spring 2000 administration of the survey, each college created an action plan to enhance student engagement in aspects important to that college; now that results are available from an additional test administration, that process will be extended and strengthened. In sum, NSSE represents a tool we have available and are beginning to use.

3.3 Summary of progress and remaining challenges

UMass Dartmouth has moved from talking about planning and outcomes assessment to acting energetically to implement them. Remaining challenges can be summarized under three categories.

• Achieving full institutionalization of planning. The written comprehensive plan needs to be made final, approved by the Chancellor, and presented for implementation. Activities associated with the plan must become embedded in on-going campus processes and decision-making. The plan must evolve as a "living document."
• Developing a campus culture of assessment and evaluation embedded in a process of continuous improvement. Efforts and successes at the program level have accustomed many members of campus units to embed assessment of outcomes within their activities.
• Incorporating evaluation into planning. Efforts and successes in planning have so far focused on what to do and how to do it. We will now incorporate processes of evaluation that will measure success and indicate necessary adjustments.

UMass Dartmouth substantially meets standard 2.2, which stipulates, "Planning and evaluation are systematic, broad-based, interrelated, and appropriate to the institution's circumstances." The university's strategic plan, which is about to move to final approval stages, is comprehensive and has already proven to be an important tool for managing institutional progress. Colleges and departments are engaged in planning and assessment of outcomes; some have implemented quite comprehensive structures, and others are making good progress.

UMass Dartmouth has made progress in meeting Standard 2.4 as regards verification of results of institutional planning. Standard 2.4 stipulates that "... evaluation enables the institution to demonstrate through verifiable means its attainment of purposes and objectives both inside and outside the classroom." As stated just above, work in institutional planning has so far paid most attention to what to do and how to do it. Incorporating processes to verify attainment of planning purposes and objectives is an important finalizing step that UMass Dartmouth must take.

UMass Dartmouth has progressed significantly toward meeting Standard 2.4 as regards assessment of learning outcomes (the requirement "to demonstrate ... attainment of purposes and objectives ... inside the classroom"). Some units and programs are actively doing learning outcomes assessment and the others have developed criteria and a methodology and are about to implement them in practice.