To: John A. Sebert, ABA Consultant on Legal Education
From: Peter W. Martin, Legal Information Institute, Cornell Law School
Date: 10/23/2001
Subject: A Report on the LII's Two Multi-Law School Courses Conducted via the Internet in 2000-2001
In 1996 we (Cornell's Legal Information Institute or LII) invited a number of law schools to join us in an Internet-based course. Three accepted: the University of Colorado, Chicago-Kent College of Law, and the University of Kansas. For three successive years students at those institutions and Cornell studied "Copyright and Digital Works" with me and each other. The course made use of "off the shelf" Internet software (Web pages, Web-based conferencing, desktop video conferencing, and e-mail) and succeeded in adapting the law school interactive paradigm to this quite different educational environment.
Begun before the ABA's 1997 "Temporary Distance Education Guidelines" the course, nonetheless, fit within them being: a) "disseminated from one law school and received at another law school" (see section 1 of the guidelines), b) highly interactive (see section 4) and c) based on commercial grade technology (see section 5). In diverse settings including two CALI conferences (1998 and 1999), three AALS annual meetings, an ABA-sponsored distance education conference (Nov. 1999) and meetings of the section's Technology Committee the LII reported on this three-year experiment as it proceeded.
In the spring of 1999, we concluded that, rather than continuing to evolve the initial model further, we should take what we had learned about available technology and appropriate pedagogy, pause a year, and create a fresh pair of distance learning courses. In doing so, we hoped to break free from the weekly “real time” video conference sessions that were central to the original model. We carried through with that plan and did, in fact, offer two totally asynchronous distance learning courses last year (2000-2001). This report summarizes what we did and how and concludes with several tentative conclusions emerging from this most recent phase of the LII's ongoing exploration of distance learning in law.
The LII's first on-line course focused on an area of advanced copyright law. The 2000-2001 offerings were: 1) Introduction to Copyright and 2) Social Security Law. The former is a high demand area that many schools are forced to lump together with other topics of intellectual property in a survey course. The freestanding course we offered gave students with strong interest in the field an opportunity to go deeper, while allowing the participating schools to reconfigure their intellectual property offerings. Social Security is, by contrast, a subject that receives negligible curricular attention in U.S. law schools despite great importance and legal complexity. It is one of a host of important legal topics around which a critical mass of student interest and faculty expertise often cannot be found within a single institution. Courses covering such topics are, in our judgment, prime candidates for a distance learning structure that enables schools to pool teaching resources and students.
Both Copyright and Social Security Law are upper-class electives. Although normally they carry no prerequisites, their position in the curriculum assures that students enter with a solid understanding of the types of legal materials that frame the respective areas (statutes, regulations, appellate decisions), with skills of study and analysis developed during least one full year of law school study, and with a command of legal concepts and vocabulary that can be deployed to gain understanding of these new fields. As taught in most schools neither course focuses significantly on the development of professional skills or involves experiential learning in the way a clinical course does or an evidence, procedure, trial practice or negotiation course may. Instead they are organized around their respective legal domains. Whether taught in a classroom or on-line, they characteristically aim to build a solid understanding of the salient features of the respective fields, plus a sense of how to approach key issues from several important lawyer perspectives such as those of advocate, counselor, advisor, arrangement-maker, policy analyst or critic. Since neither area is static, topics of proposed or imminent change regularly prompt analysis of contending policies and political forces.
In short, this second round of distance learning experimentation continued the LII's original, quite conservative concentration on on-line instruction as a means of providing greater content depth or breadth in the upper class curriculum through the sharing of teaching resources and students. It neither ventured into the first year learning experience nor into a broader range of upper-class course types. The principal difference between these courses and their predecessor consisted of the substitution of fully asynchronous components for “real time” exchange. This change freed instruction from the scheduling constraints inherent in assembling students at the same moment across multiple time zones and academic schedules. Less obviously, it allowed larger enrollments without loss of interactivity or accountability. This, in turn, made investment in reusable multi-media content economically feasible.
A total of seven law schools accepted the LII's invitation to join in last year's experiment – Arizona State, Chicago-Kent, Kansas, Rutgers-Camden, Rutgers-Newark, Seattle, and Vermont – with four signing up for each course (Chicago-Kent participated in both).
Were distance education arrangements like this standard practice, with most law schools participating as providers and receivers, the basis for a school's decision to participate or not as to either of these particular courses would have been straightforward. Assuming adequate assurance about the quality of the materials and instruction, the issue would reduce to considerations of curricular fit, faculty deployment, and budget – e.g., does the course add an important subject the regular faculty cannot cover (Social Security) or provide desirable flexibility and increased options in an area of strong student demand (Copyright) at an acceptable cost?
Because interest in distance education is high and experience rare, there was a further reason for schools to join in. The experiment offered an opportunity to observe carefully designed and executed distance education at first hand. Few, if any, other law schools have experience in Internet-based legal education comparable to the LII's. Participating schools were invited to designate one of their own faculty members as an "auditor/observer." Every effort was taken to open up the process of course construction and delivery. Schools were encouraged to use interviews, questionnaires, and other means to evaluate student response and educational effectiveness. In short, participation provided a means of experimenting with distance education for schools that had not yet done so and of exploring an alternative model for schools that had.
In order to share the results and compare our experience with that of others, the Legal Information Institute ran a workshop on distance learning course design during the summer of 2001. The weeklong conference drew faculty members and technical staff from a broad diversity of institutions. Schools that had participated in either one of the 2000-2001 LII on-line courses were assured of inclusion and contributed importantly to the proceedings (available on-line at: http://www.law.cornell.edu/background/distance/workshop/).
As with the LII's earlier distance learning venture, all participating schools retained responsibility for course registration, exam administration, and related logistical matters. Students registered not with Cornell but with their home institution. Grades and credits were local. Participating schools were also responsible for front-line technical support for their own students (with LII back-up), for assuring that their students had adequate computer resources and Internet connection, and for providing a meeting room suitable for local discussions among enrolled students.
In order to take the course students had to have regular access to multi-media capable computers with sound and Internet connections capable of delivering streaming audio reliably – either in the law school or elsewhere. As previously noted, participating schools were encouraged to designate a local faculty member to be included in all course communications. In short, as was true of our earlier distance education arrangements, these courses were, to use the language of the ABA temporary guidelines, in all critical respects "disseminated from one law school and received at another."
The Legal Information Institute's responsibilities included: preparation and distribution of course materials (free in digital format to the students), instruction, performance monitoring (the on-line analog of attendance), student evaluation and grading. Participating schools paid a fee of $500 per student per course.
The basic components of this latest LII distance learning model included:
Because the LII's earlier distance learning course included a weekly video conference it could not be fit completely within a single academic term. Inconsistent start times and vacation patterns among schools required distribution of those real-time class meetings across two semesters. The current model, being more flexibly constructed, did not require all students to be “in session” at the same time. This permitted each course to be conducted within a single term. Copyright was offered in the fall; Social Security, in the spring. The precise beginning and ending dates were set to correspond to the schedules of the participating schools. Final exams were taken during each participating school's regular examination period. In order to allow full use of the digital materials with which the courses had been conducted at this critical point, the exams were administered on a "take-home" basis in schools that did not otherwise permit students to use computers to write their exams.
Our previous model of distance learning worked only with relatively small numbers – e.g., eight students per school and 32 total. One important question we sought to answer with the new approach was whether it could effectively accommodate larger enrollments, comparable to those common in specialized law school courses taught conventionally, without sacrificing interactivity or other qualities important to effective learning.
Working closely with the participating schools we achieved enrollment figures that provided that scale. Ninety-five students took the fall term Copyright course; sixty completed Social Security Law.
The form of distance education most widely practiced by law schools to date represents the simple extension of conventional classroom practice through high-end videoconferencing technology. Classrooms at more than one location are linked to permit a teacher at one of those locations to lecture or conduct more elaborate presentation to students who are assembled at the same time in other locations. With additional investment in technology infrastructure those remote students can participate in discussion with the teacher and each other. The principal advantage of this mode of distance education is that it requires very little adjustment of working patterns or expectations on the part of either teacher or student. It can be used to create highly diverse collections of students (students gathered in classrooms in different countries, for example) and can link faculty members with students they might otherwise be unable to teach.
Major drawbacks to this form of distance education include high-cost at both the sending and receiving end and its requirement that faculty and students assemble in "real-time." The latter can become increasingly problematic as sites are linked across time zones.
The LII's current distance education courses rely on less costly technologies and embody patterns of instruction that make substantial use of asynchronous exchange and pre-programmed instructional materials. The choice reflects a conviction that, long term, this direction holds the greatest potential gains from networked digital education. Last year's experience only strengthens this belief.
For law courses of this type, i.e. content-defined and focused on a statutory domain, and with students already well grounded in the study of law, the model appears effective. The LII's 2000-2001 offerings generated educational outcomes that compare favorably to those realized in similar courses taught in a conventional, classroom-anchored mode.
Some might argue that in light of the widely observed tailing off of attendance and engagement among upper-class law students that constitutes too low a benchmark. Unquestionably, our ambition is to achieve substantially better outcomes (at no greater cost) through on-line methodologies, and our tentative conclusions, set out below, encourage us to believe that we are headed down that path. But in any discussion about regulatory and administrative structures that privilege classroom instruction (as the current ones do), the effectiveness of conventional teaching methods inevitably and appropriately becomes the initial basis for evaluation.
The students enrolled in these two courses represented a greater range of language and analytic skills, work and life experience, and facility in doing “law student work” than a teacher is likely to confront in a single law school student body. Since some of the participating schools had part-time divisions, the mix of students in both courses included significant numbers who brought directly relevant work experience to the exchange. Taking account of that diversity, I judge the quality of student work product I saw through the two terms (the weekly problem submissions, on-line discussion contributions, mandatory mastery exercises, and final exam) to be of very high quality. Measured in terms of: 1) understanding and mastery of course content, 2) sustained engagement, and 3) learning from one another, the course outcomes were, overall, better than I would expect to achieve with the same students meeting thrice a week through the term.
With a classroom-anchored course culminating in a final exam, there are few reliable mechanisms for monitoring individual student progress during the term. Attendance may or may not be effectively tracked. Class preparation may or may not be audited by periodic queries directed at non-volunteers. Even with the most rigorous application of “Socratic” teaching the large upper-class course provides plenty of cover for students who opt for a “wait and then cram for the exam” approach. In contrast, on-line teaching methods enable a teacher to be far more attentive to the progress of individual students. The model of asynchronous instruction represented by the LII's 2000-2001 courses included weekly progress expectations and four mandatory progress checkpoints (the “mastery exercises”). Placed in a work environment that logged student contributions, it facilitated prompt intervention when any student fell behind. As teacher I experienced an unfamiliar level of confidence that I was detecting student difficulty or simple procrastination in time to make a difference.
With a conventionally taught course, the overall work load is crudely measured by the minutes per week of class time. There are three credit hour courses and four. Inevitably some three-hour courses impose more demands on students than others, but, over time, students themselves police excess in one direction, faculty colleagues and academic administration, extremes in the other. For an asynchronous on-line course, with no shared clock running on any part of the student-teacher interaction, there is, at present, no such convenient metric. Each of the presentation or problem modules comprising these courses did carry an explicitly stated “run time”. But summing up these times yields a useless number, for several reasons. First, most students exercised the control over delivery offered by the on-line environment to pause, reflect, take notes, and repeat. Consequently, completion of any 20-minute module could easily take twice as long. Total time spent reading, reflecting on and responding to messages in the on-line discussion area or composing a problem submission can only be the subject of speculation by course builder and teacher.
So long as such courses remain the exception rather than the norm, however, their contents and work demands can be benchmarked against conventional ones. Both of the LII on-line courses covered exactly the same ground as their classroom-based equivalents at Cornell.
Students in both courses were encouraged to fill out a final course questionnaire, on-line, during the week before their final exam. The response rate was 30% in the fall, 38% in the spring.
To begin, the students (who were predominantly in their last year of law study) were asked to compare the on-line course they were completing with "other specialized law school courses with comparable credit." Specifically they were asked about: (1) the total time and effort required, (2) their own success in mastering the material covered, and (3) the quantity and quality of teacher feedback and amount of discussion with teacher and other students. Answers were consistent across the two courses with a strong majority of students reporting that they worked harder, achieved comparable or greater mastery, and experienced more feedback and exchange than in a classroom course of similar content. (Since the Social Security course came second, it benefited from a few lessons learned during the first term and, to the extent there were discernible differences in student evaluation of the courses, it received slightly more favorable ones.)
The challenge of disaggregating the range of activities carried out during standard classroom meetings into presentation modules, interactive exercises, problems for analysis and submission, and on-line teacher-student discussion forces an unusual degree of attention to pedagogical ends and means. Done properly, the resulting asynchronous components allow students to organize their course work for more successful learning. The fixed length class at a standard meeting time imposes significant burdens to which most law faculty members and students are blinded by familiarity. The duration is often too long, sometimes too short. In terms of student readiness and attention, the moment is arbitrary and, therefore, frequently the wrong one.
By contrast, students taking these on-line offerings had enormous flexibility in how they fit the multiple course elements into their weekly schedules. Such control over the exact time and place of their learning was, for the students, the most highly valued feature of the asynchronous course architecture. Not only could students to take up a given module when they were ready and able to focus, they could run it, pause, take notes, and return to puzzling points. In numerous ways they could exercise a degree of control over each step of the process quite impossible in a real-time classroom session.
Course questionnaire results indicate that more students than not took advantage of the capacity to pause and replay the course presentations and also to block and copy the presentation outlines and visited text (e.g., statutory provisions) in the course of note-taking. As one student explained: "What was great about the [format] was that I was able to go through the [presentation] once but pause many times. I would write what you said and then stop .... Then I would look at the statute or regulations and highlight it and make notes in the margin."
In addition, modularity has a direct bearing on future reuse, course maintenance and other important elements directly related to the economics of this form of education, a topic addressed below.
In faculty workshops and other settings where these courses have been described and demonstrated, one question that recurs is why we chose audio rather than video for the “teacher presentation” components. The reasons are numerous.
To begin, there is a stark cost-benefit difference between the two. While the educational gain from using video can be enormous when the medium is being used to show process or action – whether the topic be marine biology, migration trends of the past century, volcanic eruption or cross-examination at trial – the educational gain from adding a head and gesture to the teacher's voice is minimal. No doubt some psychological value flows from students being able to visualize their professor, but that can be realized through a short video introduction.
On the cost side, the gap is enormous. Video imposes several different types of added cost. Most obvious is the greater expense of initial production. Students bring expectations of broadcast quality to video material. Creating video content to that standard is more expensive than first-rate audio by several orders of magnitude. Furthermore, whatever assumptions one makes about the ongoing rate of course revision in successive years, even at levels as low as 10-20% per year video, being far more complicated is, therefore, more costly to maintain.
Bandwidth is a totally separate matter. The streaming audio, multi-media technology used in the LII courses operates quite reasonably over a dial-up Internet connection. Streaming video does not. Its use requires students to have more capable and more expensive network connections, and it also obliges the offering institution to have greater serving capacity.
Most law teachers create in relative solitude. They write, they prepare and deliver their courses by themselves. Use of teaching assistants is rare. Effective though they may be in live lecture and discussion formats, few are comfortable and as skillful before a camera.
For these and other reasons, a mode of multi-media course production that begins with a microphone attached to the law professor's office computer is accessible to many more teachers than one requiring use of a studio and video crew. With current authoring tools, high quality audio can be prepared, edited, and revised by a faculty member, alone, in his or her office. Software tools designed to allow presenters of all kinds to prepare audio files to accompany their PowerPoint slides can be readily adapted to the creation of presentations that refer to a wider range of Web-based material. With a set of course materials on-line, the teacher can speak about a statutory section or passage in an assigned case and have the very text automatically loaded in the student's browser as it is being discussed. The student can, in turn, pause the audio in order to reflect on the text, copy portions into his or her notes, or to follow hypertext links that connect to related material, as for example, another statutory section defining key terms or qualifying its apparent meaning.
Within a modular architecture, such content can be assembled in different combinations and configurations. It can, in successive years, be altered by adding, subtracting, or substituting new audio and textual material, without the need to rebuild from the beginning.
The interactive features included one that drew near unanimous praise, the set of four "mastery exercises". These mandatory problems, placed at the end of two to three weeks worth of material, called for students to submit a few paragraphs of analysis, via a Web form. Each problem was posted a week before its deadline. Forty-eight hours after the deadline the class received a generic (i.e. non-individualized) feedback memo. It set out my view of the issues and responded to common errors or confusions revealed in student submissions. A typical student reaction to this course component read: "I thought the mastery exercises were useful and the feedback was very helpful too." Another student, perhaps more candid, wrote: "The mastery exercises were a necessary evil. I am thankful they were included insomuch as it is too easy to fall behind. Those exercises forced me to keep up with the material and review it again and again until I found the appropriate answers." Yet another exclaimed: "Finally a law school class where a question is not answered with another question."
Also popular were the interactive exercises that allowed students to gauge their level of comprehension following the presentations and readings on a topic. This is one dimension in which several expressed a desire for more: "The pop up windows for the illustrative questions were great. I wished they popped up for each question."
The on-line class discussions had both fans and detractors
with the obvious base for comparison being discussions in a conventional class
of similar size. Objectively several things seem clear to me. Including the
forms which asked every student to submit a question or take a tentative position
on a hypothetical problem before the related on-line discussion began many more
students were involved in the exchange throughout the course, topic by topic,
than I have ever been able to bring into "real time" classroom discussion.
The evident degree of reflection and level of discourse were high and I observed
more frequent introduction of personal experience and references to material
outside the assigned readings, including current events. However, just as some
students found this a less inhibiting venue for "discussion" than
a classroom, others exhibited at least comparable reluctance to "speak,"
perhaps in part because the class included others they did not know, from different
law school communities. Some reluctant posters said they liked the discussion
environment, nonetheless: "I like the discussion area. I like to read
all the messages even though I didn't put anything there.
:-)"
Most of the students were pleased with their on-line instruction. A number pronounced it their best law school learning experience. What were the complaints? Anyone who has taught an upper class law school course can imagine many of them. To some my voice was soothing, to a few it was monotonous. The expectation of involvement struck a handful as "unrealistic." Said one: "I have never been in any class where EVERY person has something to say on EVERY topic!"
Based on a review of the full range of student feedback, I have concluded that most individual student frustration and dissatisfaction can be traced to one of three sources: 1) technical problems, 2) specific expectations of what an on-line course would or should be that were not fulfilled by this one, and 3) the challenge of dealing with digitally-delivered course readings.
Most students reported few or no technical problems. Those with the worst experience fell in one of two categories. They were either at a law school plagued with frequent network difficulties (true of at least one of the participating schools) or they were relying on an Internet service provider (ISP) with inadequate capacity, most frequently AOL. Learning from the fall term course, we emphasized at the outset of the second one that students should, as a test, run the introductory unit of the course under the conditions (computer, ISP, time of day) most likely to prevail later while they would be taking the course.
Upper class law students know what to expect in a classroom-based course. Currently at least, those taking an on-line course have far less certain ground for anticipating how the course will be conducted or how they will, as individuals, respond. To deal with this source of potential difficulty, we placed materials at the beginning of each course that explained and also modeled how the course would function. This is a second area in which our second course benefited from experience with the first.
On-line courses can be taught from standard commercially published casebooks. Concord does so routinely. A separable feature of the LII's approach to date has been the use of original course materials that, being free of the license constraints, can be delivered on-line along with the rest of the course. This approach permits tight integration between the readings and the on-line instructional materials. With a full set of course materials on the Web, our presentations can and regularly do link to portions of the assigned readings. The same is possible with comments posted in the on-line discussion area.
On-line delivery need not mean reading from the screen. For both courses the readings were offered in parallel versions: a browsable format and also downloadable word-processing files ready for printing. The range of choices and the consequences of some of them (i.e. the bulk and, at some schools, the cost of printing out the full course materials) surprised a number of students. The complaints on this score were significantly reduced in the spring, probably also a consequence of greater clarity in the course introduction about what to expect coupled with specific advice about how to manage the readings.
Institutions and teachers need to conceive of the preparation of on-line courses as being much more like writing a book than like teaching a class. The time, level of commitment, and need for attention to detail required to create the first version of a multi-media on-line course can be justified only if viewed in book-like or long-term investment terms.
When the course is constructed of modular and reusable elements that investment can pay attractive dividends, but only when the course is offered to successive cohorts of students. As it is offered for the second and third time, requirements of teacher time and effort reduce to those entailed in revising the reusable material and interacting with, monitoring, and assessing the students.
Encouraged by last year's experience we are repeating both courses this year. Since they were built around reusable modules, designed with an eye to revision, we have, as we planned, been able to make updating and other changes selectively (without starting over from scratch).
The course architecture also contemplated re-assembly with appropriate additional material for different student populations. To explore this process and its potential, we have adapted last year's Copyright Law course to produce an offering that introduces the principal issues and features of the field to university students in such areas as journalism, art, music, and computer science. This non-JD course is being tested at Cornell during the current term in anticipation of wider distribution.
The Social Security Law course will again be offered to law students during the 2002 spring semester. Participating law schools will include some of last year's group joined by one or more others.
Both LII courses, as offered last year, remain at the institute's Web site. Unlike conventionally taught courses where only the residue following completion rests in notes and memories, they can still, months later, be inspected in full.
While the sites are not open to the public, any member of the Council, its staff or committees are welcome to visit them. The respective URLs are:
http://www.law.cornell.edu/copyright/fall2000/ [Copyright]
http://www.law.cornell.edu/socsec/spring01/ [Social Security]
The materials are, in differing degrees, password-protected. For anyone seeking an overview, the best place to begin is with Social Security site. It allows access, without password, to the course syllabus, readings, multi-media environment, and introduction. To proceed further with that course or to inspect any of the Copyright site requires a guest password. It is available upon request. Anyone wanting the password for either course or further information about them should feel free to contact me directly, by phone (607-255-4619) or by e-mail (martin@lii.law.cornell.edu).
* The presentation component used audio rather than video because of our conviction, confirmed by experience, that the substantially greater cost of making and revising video materials is not warranted so long as the principal content would be a "talking head." For more on this point, see IX.D. infra.