December 2009 - C-Print®: Technology Research for Deaf and Hard of Hearing Students |
|
C-Print®: Technology Research for Deaf and Hard of Hearing Students
Back in the 1970s deaf students were aided in classrooms by sign language interpreters. As an increasing number of deaf and hard-of-hearing students entered general education classrooms, schools began to provide in-class notetakers, who delivered their notes later in the day. In the 1980s computer technology complemented the skills of court reporters who used stenographic shorthand to produce a word-for-word display of what speakers where saying as they spoke. In the 1990s an easier to learn computerized word abbreviation system was created that enabled a large number of service providers to produce captions so that students could understand what was happening in class. Today, service providers are aided by tablet technology similar to that currently used by newspaper and magazine publishers, to produce state-of-the-art speech-to-text displays for deaf and hard-of-hearing students. Many of the advances of the past two decades bore – and continue to bear -- the C-Print label and were inspired by Mike Stinson. The C-Print Team Speaks “My work in helping to create new captioning methods and software has been driven strongly by my personal experiences as a deaf person,” he says. “I was mainstreamed and was in regular classes all of my life; I was never in a school for the deaf. I didn’t learn sign language until my Ph.D. was completed. I struggled with communication.” Dr. Stinson attended the University of California Santa Barbara for his freshman and sophomore years “and almost flunked out.” He had studied diligently, he says, “but I did not understand back then that the reason I wasn’t doing well was that I didn’t hear enough of the information. My hearing loss was progressive, so my hearing was becoming steadily worse. I thought maybe I wasn’t smart enough.” Over time, he adds, “I learned to understand that lack of communication was the culprit, not my intelligence. My aim since the very beginning has been to try to find ways to help deaf and hard of hearing people access communication.” C-Print’s development coordinator and technology maven Pam Francis has been part of the C-Print team since 1993. Senior research scientist Lisa Elliot has been with C-Print for since 1996. Ms. Francis is an expert in speech-to-text technology, including direct service implementation, training, software development and hardware and ergonomics. She oversees the development of the C-Print Pro software and C-Print training and plays a key role in the continued development of the C-Print system. Since 1993 she has been involved in the research and development of speech-to-text captioning systems. Lisa Elliot received her Ph.D. in human development from the University of Rochester. As a member of the C-Print staff, she has been involved in the research, development and evaluation of speech-to-text captioning systems. Research captionist Anne Alepoudakis, research assistant Donna Easton, and software developer Justin Mahar fill out the C-Print research team. Housed at Rochester Institute of Technology’s NTID (http://www.ntid.rit.edu/), C-Print (http://www.ntid.rit.edu/research/cprint_home.php; http://www.ntid.rit.edu/CPrint/) has been funded since 2000 by Steppingstones of Technology Innovation for Children with Disabilities grants awarded by the U.S. Department of Education’s Office of Special Education Programs (OSEP). These grants are awarded to promote the development, implementation and evaluation of innovative technology tools and practices for students with disabilities. Most of C-Print’s Steppingstones grants are rooted in implementations associated with C-Print Pro software (http://www.ntid.rit.edu/CPrint/product.php), which is specifically designed to provide C-Print speech-to-text services and allows service providers, called C-Print captionists, to input text using a computer-based keyboard abbreviation system. According to Ms. Francis, the first version of C-Print in the early 1990s utilized commercially available software, including Productivity Plus, a word abbreviation system, and WordPerfect. “We actually used three or four connectivity software packages,” she explains. “This was back when we wanted the process to move from computer to computer, from captionist to student. None of the software packages met our needs.” Adds Dr. Stinson, “The commercial software we used until 2000 was not really intended for the service we were providing. The first Steppingstones grant funded the development of the first C-Print software, which was created to deal with the limitations that we’d been experiencing for seven years.” Ironically, Dr. Stinson continues, “We didn’t originally intend to create a software package, but then, there was this very specific need and we felt that creating something of our own was more reasonable and desirable in terms of better serving future student needs.” In describing the start of this program of research and development on captioning, Dr. Stinson noted, “I did a study when I first arrived at NTID in the late 1970s that compared how much deaf college students here remembered when they read a passage versus when they received the same information via a sign language interpreter. I found that the deaf students were able to remember more information through reading than when they saw the sign-language interpreted version. That led me to think that if we could devise a way to provide a printed display for students in the class there might be many deaf students who could benefit. “ During the 1980s, he continues, “we used a stenographer system, which was expensive and for which it was very difficult to get competent stenographers. The students, however, thought the stenographer-based system was more beneficial than interpreting. Their feedback was overwhelmingly positive. What we’ve developed with the C-Print system, thanks to the Steppingstone grants, is a less costly system for which it’s often easier to locate service providers than it was with the stenographic system.” We also feature members of our Knowledge Network. Please share this newsletter with other organizations, families and professionals who may benefit from it. We invite you to visit us at http://www.fctd.info. We welcome feedback, new members and all who contribute to our growing knowledge base. “Progress Has Been Made… An Interview with C-Print research team members The research path from the early days has been a hard but steady climb. While novel for its day and beloved by 1980s students, the stenographic system based on court reporting does not does not take advantage of certain new technologies that can improve the communication access and learning of deaf students. For example, steno-based systems do not display graphical information, or enable students to easily adapt the display to meet individual needs or add their own input to the information being displayed. In contrast, the tablet PC-based speech-to-text system being refined by the C-Print team in its 2007-10 Steppingstones grant has all these features, C-Print researcher Lisa Elliot says. “Nothing Is Left Out” Mastery of the craft of stenographic court-reporter-style transcribing, which employs abbreviations, requires 3-4 years of study, she explains. “The equipment was expensive. From an educational perspective the court reporting form of captioning was far too costly for most schools to provide. Mike wanted to find a different way.” “I still have samples of stenographic transcripts from the 1980s,” Dr. Stinson says. “They are very long and very hard to study. We heard a story from a student who used C-Print in secondary school here in Rochester and then went to a college where only a stenographic system was available. The student was so frustrated with the notes that she gave up using a stenographic system and went to using an interpreter.” The speech-to-text display produced with C-Print software represents the spoken dialogue of the classroom using a software application called C-Print Pro™. The text can be displayed simultaneously to one or more students in different ways, including additional computers (laptops) or display monitors. The captionist includes as much information as possible, generally providing a meaning-for-meaning (not verbatim) translation of the spoken English content. Changes in delivery technology aside, C-Print has always been based on a dictionary comprised of complete words accompanied by corresponding abbreviations. Captionists learn a rule system in order to type abbreviations instead of complete words. Says Dr. Stinson: “They can’t memorize every word so they use the rules. With Pam overseeing the evolution, we have continued to revise and improve the rules and the abbreviation system.”
Dr. Stinson and his team began the transition from stenography “with basic computers, using basic word processing programs,” he recalls. At first, Dr. Elliot remembers, “we used off-the-shelf computer programs - word processing programs, abbreviation programs and a program that allowed the computers to communicate with each other.” “When several programs were used, it was difficult for captionists to operate all of the applications simultaneously, and it was difficult for the project team to maintain continuity with changing software versions. We eventually decided to write our own application that would combine aspects of all three programs to create a single, smooth program. The C-Print Pro software (http://www.ntid.rit.edu/CPrint/product.php) has been developed with the aid of RIT students, who work with us through the RIT co-op program or as part-time employees while they’re in school.” Keeping Up with Technology: the Tablet PC She adds: “We are not only doing product development or providing a service. We are conducting the research that shows the effectiveness of existing applications, ours and others’. We’ve been doing this for a long time and have developed the technology and conducted the research that has allowed others to move forward technologically.”
Those ways, she explains, include real-time speech-to-text services with graphics and real-time notetaking. The speech-to-text with graphics option includes teacher-student classroom exchanges; that is, the student views a real-time display of the text of what is being said, just as with traditional C-Print, and in addition, can view graphics produced by the provider. With the real-time notetaking option, the student is able to view in class what is written as it is written by a notetaker.” This new form of notetaking is quite different from traditional notetaking. “Students using traditional notetaking services often received their notes from another person paid to take notes in their classes. That person took notes on four-part carbon as she sat in the rear of the room. Students received those notes at day’s end. The notetaker did highlighting and filled in worksheets. Student and notetaker fulfilled their respective tasks independent of each other. This was a very labor intensive system and very hard on notetakers’ hands. It can be frustrating too, especially if the carbon doesn’t work properly.”
The Study The 2007-10 Steppingstones study is comprised of three groups of students: the real-time notetaking group; students who receive speech-to-text with graphics; and a control group. When the C-Print team applied for Steppingstones funding from OSEP, team members identified the following research questions:
“In regard to the last question,” Dr. Elliot notes, “what we’re looking at are student characteristics such as level of hearing loss, reading ability and the ways students employed the technology. There are questions about student motivation. We also look at teacher practices in the classroom and at student familiarity with computers; some students are more comfortable with computers than others.” The study segment of the current grant encompasses 90 students in several metropolitan areas nationwide. “Some of the students may have multiple disabilities, but whether or not they have multiple disabilities is not a contingency for participating in the research. All the student participants have been diagnosed as deaf. Some are hard of hearing. Some are profoundly deaf. Some use hearing aids. Some have cochlear implants. Some use FM systems or interpreters. The support services they receive or the kinds of assistance they have don’t matter. Every student who is in a mainstream setting can participate.” According to Dr. Elliot, the 10-week period of study is modeled after the traditional 10-week school marking period. Students in the technology trial groups use the tablet PC technology in just one science, technology, engineering or math (STEM) class for five weeks. Classroom teachers evaluate student progress made using the new technology. Classroom teachers also evaluate student progress for five weeks before or after the trial period. For 10 weeks the control group receives no technology and employs only the supports used previously. Control group students are also evaluated by their teachers. After that period the control group gets a one-week trial, but no data is collected from the group. Students in the study participate in an experiment in which they all view the same simulated lecture. According to Dr. Elliot, “they take a pre-test, view a video, then they do a post-test. The way they view the video depends on which group they’ve been in, because we’ve tried to simulate a class experience for them with the support technology that they actually used. If they were in the real-time notetaking group during the regular trial, then there is a screen that shows the teacher and the lecture and a second screen that replicates the notetaking that is taking place. It’s the same arrangement with the speech-to-text with graphics group. The third group – the control group -- has an interpreter.” The objective, she notes, is to recreate the students’ classroom experience and to compare student performance on the mini-lesson assessment in each of the three groups. Phases of C-Print Research Focus groups are conducted with students, parents, service providers, and teachers. “We bring in consultants and others with technology backgrounds to provide us with their input.” Trials are then conducted. “We test [alternative] options and constantly get feedback in order to improve them. We conduct 15 trials for the typical project, inputting continuous improvements as the trials progress.” All phase one activities are conducted in the Rochester, NY area.
“We include teacher ratings and student opinions as well as objective measures to ascertain how much they are learning with the option they are using.” These strategies, he explains, “enable us to develop a more quantitative assessment of how much the students benefit from the technology. We also take measures to address fidelity of implementation, i.e., how do the programs implement the services with their service providers.” Fidelity of Implementation Because members of the small C-Print research team are unable to deliver services everywhere at once, fidelity of implementation, according to Dr. Elliot, must provide answers to the following questions:
“We’ve created brochures and thought hard about these approaches. We conduct orientation meetings with site staff to determine what should be said in a meeting to introduce a trial with the technology and who should attend. Details in regard to implementation of a technology in a particular classroom are settled before we enter a classroom.” Randomized Design Challenges Fortunately, she adds, “we’ve been successful in gathering students to produce statistically significant results because our reputation is a good one. We’ve worked with some programs for years. Sometimes it can take longer to assemble the student participants than we originally planned.” When a randomized design is required, the volunteer participants are not told which group they’ll be in – it could be the control group. “We devised a way to reward the students who had to be in the control group; they received a free week to use our technology. After the data is collected we return to the school and provide the participating student an opportunity to sample the new technology. Unfortunately, a week is not a lot of time,” Dr. Elliot comments. “It’s a reality that when the study ends, the new equipment departs with the researchers. All of our research and development is based on feedback from students, teachers and service providers. I observe how the technology is used and then talk to participants about their experiences. All our changes are based on that feedback. It’s an iterative and customized process.” “The problem is that when we test the software in a live classroom and the students like it and need it, removing it can be a wrenching experience for all. Often the administrators, parents and students volunteer to participate because they think the tool will be beneficial, and it often is. And then what? That’s the downside. The upside is that we are careful to develop something that will work and will eventually get to the schools.” Sometimes, she adds, “administrators may be reluctant to allow us to come into the school because if our equipment is effective families will want it and the school will have to provide it. We’re not in this to make money. We’re not in this to make a profit. We’ve created the technology so it can be provided to schools and individuals at low cost. Some administrators are reluctant to get involved because they don’t want to make an implied commitment to the future.” The C-Print Team: Ongoing Enthusiasm and Varied Skills Are Hallmarks The team’s development effort, she continues, has been tasked to software engineering and computer science students from RIT. “The RIT connection is a strong positive. This is training and preparation for these students to move on to other software development activities, to graduate and find other work.” Often, though, she adds, “their work with us has encouraged them to seek out meaningful, non-commercial work. Because our project encompasses aspects of software research, development, and implementation the students who emerge from this project are usually well-rounded. This is consistent with the RIT tradition of preparing students for valuable technology careers and is well-received by federal funding agencies who like the idea that we are creating a learning environment for the students who are going to move on.” During the 16 years since their first Steppingstones grant, the C-Print team has trained more than 1,500 captionists at RIT and elsewhere. According to Ms. Francis, about 25% of the practicing captionists provide services in middle school and high school settings. Evolving with the Technology The centerpiece of C-Print’s current 2007-10 Steppingstones grant, the tablet PC, is especially effective for students taking courses like math, technology, science and engineering in which information is presented by alternative means that include graphs, illustrations and formulas. “Typing, which suffices for courses like English and social studies, does not adequately capture this information,” Dr. Elliot declares. Students using an interpreter must keep their eyes on the interpreter while trying to absorb other visual information. “It’s very difficult for them to look at the interpreter and the overhead projector or the board. If they move their gaze away from the interpreter they lose the information. We’ve been able to bring some of that information into the same visual space by using the tablet PC.” “Notetaking Is Active” Previously, she explains, deaf students had a notetaker who sat in the rear of the room and delivered the notes to students later in the day. “Our software enables students to interact with the notes,” she states. The C-Print software, she explains, enables students to highlight the information transcribed by the captionist as it is being transcribed. “Students create a notepad on which they can copy and paste. They can write questions for themselves. Because they are working on a tablet, students don’t even have to be able to type; they can draw with a stylus. They can create symbols of emphasis like stars or asterisks. We have learned that students who are able to manipulate the notes, not just passively read them, tend to do better on quizzes and tests.” Another C-Print software advantage, she points out, is that it empowers two-way communication. “We’ve included ways to allow students to communicate with the service provider via an instant messaging (IM) system. If the student is unable to voice, she can type/write a note and the provider can voice the student’s question or comment to the class. It enables the student to have a more private conversation with the provider, perhaps about a word or phrase. There’s communication support as well as access to information. It’s often very difficult for deaf students to participate the same way as hearing students. The software enables them to do that.” According to Pam Francis, “the basic C-Print software is straightforward. There are two applications that communicate with each other. The information that is produced by the service provider is sent to the client. The two computers communicate via the software. The basic connection protocol is not fancy. Most computers have this protocol today. The computers do not require a broader network in order to be connected. They can be connected via an ad hoc connection.” Introducing Complementary Technology A mobile version of the software can be used via a cell phone, she notes. “There might be a need to have text in an environment that does not necessarily have a traditional network access. Our companion National Science Foundation project calls for us to develop a mobile version that will enable deaf and hard of hearing students to use text in an environment that does not have local network access, like a field trip, where students can use a cellular Internet connection.“ A recent trial of the tool, she says, involved a visit to a local museum by a client with low vision. The results, she notes, could have benefits for deaf and hard of hearing students as well. Explains Ms. Francis: “The student used some capabilities built into our client application that would allow her to have a larger font. She could view the captioning on a cellphone, adjusting it to a setting appropriate to her visual needs. This assisted her in a number of ways. For example, often museums are dimly lit to preserve the artifacts, so they lack the lighting that would enable an interpreter to work effectively. This device allowed the student to read the captioning despite the museum lighting. That was a situation in which we were able to use the cellular network. We’re still using the Internet but we’re using it over the cellular network. A campus network is equally effective and can be used instead of a cellular network. We use IP addresses, which broadens our capabilities.” Another example involved the use of the remote during a field trip to a local phone company. “We found we could use the device in an environment that was very noisy. We have a Bluetooth headset on a speaker that has noise canceling capability. The speaker delivers the information over a regular cell line via a voice connection to the captionist. The information is transmitted to the student on the student’s phone via a cellular connection.” What was fascinating, she remarks, “was that hearing students joined the deaf student in looking at the phone display because the noise level was so high that the students were unable to hear what the speaker was saying. The text was effective for all the students.” What the C-Print team aims to accomplish, Ms. Francis explains, “is to provide a display in environments that are non-traditional, that require a little more creativity in getting the information to the captionist, who can in turn send it to student. This is achieved in real time. There’s a slight lag, the same lag that occurs in any remote situation. But it’s a version of remote that is not dependent on a local area network.” Adds Dr. Stinson: “This kind of a mobile phone works much better in this situation than having the student and captionist carry two heavy laptops on a tour. Ensuring Access to the Curriculum “I’ve chosen to deal with that question as a researcher and developer,” Dr. Stinson declares. “I’ve put my energy into leading a team to try to create something that is very useful to deaf students. That’s my number one motivation, not publishing articles, but instead creating something of value. That’s not to say that I don’t want to publish articles in order to disseminate the results of what we’re achieving or not achieving. My main objective, though, is to produce tools that can be used by deaf and hard of hearing children.” His team, he says, has much experience in advising families who want C-Print. “We often provide information to families who would like their child’s school system to at least try out the technology in order to get a sense of it. We are reluctant to tell parents that they should tell their school system, ‘You must make a full-fledged commitment to this.’ Implementing C-Print represents a big commitment. A person has to be trained and paid a salary to provide the service. This is a costly step for a school system, although it’s not more expensive than an interpreter. We understand why schools may hesitate. The grant offers an opportunity to conduct a trial, and if it’s clear during the trial that a child is benefiting from the service, it becomes easier for a school system to support that child.” Adapting to the Curriculum For example, she says, “we’re finding that schools have Internet access in their classrooms. Our computers are communicating with each other. Sometimes there is just too much traffic. If there’s an interactive whiteboard in the classroom occasionally there will be interference. The service provider has to be flexible and adapt to whatever technology crisis surfaces.” A better question, she says, is ‘Can the technology keep pace with the ideas we generate?’ For example, “the first time we considered writing on a computer was in 1998. In 1998 there weren’t many tablet PCs anywhere. We had to wait until the technology caught up with us. Can the school keep up? We try to make sure that the software is usable on a range of computers – that the technology is not a memory hog, for instance.” Culturally, she adds, “schools approach computers differently. For instance, will the school allow our computers in the building? Does the school have a printer for us to print notes on? Is it a color printer? Most schools do not use a color printer. Therefore, printing is a challenge.” What Teachers Say Many itinerant teachers work with their students outside of the classroom, she explains. “The students receive notetaking support in the classroom but because it’s an electronic file we also have the capability of printing out notes, emailing the notes and posting the notes on the web. There are different ways of distributing them, but teachers of the deaf can have access to those notes. That gives them a very good picture of what’s going on in class by providing context regarding ways to help students…[it provides] information teachers can use to work more productively with their students.” There are other benefits for classroom teachers, she adds. “Sometimes they retain the notes because the notes help them plan for the following year. The notes may also be useful in preparing a substitute teacher. If the regular teacher is absent she can obtain the notes and learn what has occurred in her class in her absence.” The notes also help teachers prepare exams, Dr. Elliot says. “This year, with so many students out with the flu, the notes can be distributed to other students in the class. The notes can also provide teachers with some insight into their own classroom performance.” Some teachers worry that the speech-to-text version will record their every word, including instances in which discipline is meted out. “Most teachers, though, once they become accustomed to it, cease to be worried.” Captionists can work with teachers to decide what will be in the final copy of the notes, she explains. “In some schools providers edit the notes.” Teachers, she adds, appreciate the ability of the tablet PC notetaking option to capture students’ writing, which enables teachers to acquire insight into the information a student is likely to retain. “This opportunity didn’t exist previously.” Occasionally, she says, the team encounters a random teacher who is uncomfortable with the presence of another adult in the class. Some teachers are distracted by a provider of real-time captioning who is tapping keys while the teacher is speaking. “Fortunately, the tapping of keys for most of us has become the background noise of everyone’s life. While there are certainly criticisms – there are concerns about intellectual property being captured – teachers for the most part have appreciated having the information notetaking provides.” The Intervention: What’s been Learned? “From studies associated with our earlier Steppingstones grants we’ve learned that C-Print is definitely a viable support service for students. Based on past studies we’ve done we know that providing C-Print to a student doesn’t hurt the student. In some circumstances use of C-Print may provide more information than an interpreter. In other circumstances an interpreter is better.” C-Print, she continues, provides access to students. “Not all students know sign language. C-Print can provide that access, sometimes the only access students have. This access was previously unavailable.” The team also learned that the notetaking capability is useful. “Students are responding positively to the graphics. We can use various colored highlighters and pens. Students find the ability to use color is helpful. Take a math class, for example. Students know that working a complicated math problem requires many steps. In some middle school and high school classrooms teachers will use colored chalk or colored markers on a whiteboard to indicate steps. We can mimic that with our multi-colored pen so a student can more clearly follow the teacher’s presentation.” Students also found various ways to code the information, she says. “For instance, if three topics are being discussed during a class a student might choose a certain-colored pen to delineate subtopics. These colors aid students in organizing the information.” The team learned that “educators expect students to know how to take their own notes, but teachers don’t always explain the mechanics of notetaking to students. We can use this as a learning opportunity for students and teachers. We can use notetaking as a way to teach students how to study. We’ve surveyed the ways that students study. We can feed that back to them and show them how to make vocabulary lists, for example. We, of course, expect that students will create study systems on their own but not all students have that capability.” C-Print Staying Power – and the Future “My involvement with text-based systems goes all the way back to 1982,” Dr. Stinson comments. “The provision of speech-to-text services didn’t take hold until around 2000. We’ve been around for a long time. And we have persisted.” Ms. Francis calls herself “a ‘closet geek’ because she loves technology. “I love working with people in training. It’s been a great challenge to move things from research to the next step. Implementing that progression requires us to bridge the gap between research and practice – a sometimes treacherous transition -- and aid others in making that transition. Even at NTID that transition is often difficult. There are more than 50 C-Print captionists here at NTID and the Institute may require more to fill the need. The need for interpreters is growing as well.” “The point is that the text is not going away. In fact, the need for it is growing. Our goal is to meet that need and evolve along with the technology.” For his part, Dr. Stinson is hoping for yet another Steppingstones grant after 2010. “I’ve written some preliminary drafts in the anticipation that there will be funding. We want to continue this work. We think there are many new kinds of situations that are not currently being addressed, such as group work and communication. Deaf students often are not participating in group activities. We’d like to do something that helps them participate.” RESOURCES ARTICLES Texting Spells Freedom for Deaf Students Students, their family members and staff from the Danforth Collegiate and Northern Secondary schools in Ontario were offered 250 Bell Mobility Motorola pagers and five years of free service for the pilot program. Over the span of the pilot program the project involved an average of 40 students, 80 parents or other relatives, and 15 staff members. Project developers saw signs of reduced social isolation for students, increased accountability from the students for their academics, improved writing and literacy skills among students, and improved communication between the students and their families.
http://nationaldb.org/NCDBProducts.php?prodID=31
KNOWLEDGE NETWORK MEMBERS National Technical Institute for the Deaf (NTID) National Consortium on Deaf-Blindness (NCDB)
The center explores novel computer graphics, advanced visualization, and human computer interface technologies, such as auditory, haptic, and multimodal interaction. These technologies are integrated with state-of-the-art advanced computation and networking, and high-end immersive visualization environments to assist researchers and industry. For more information on the center, contact:
Funding provided by the US Department of Education under grant number H327F080003 Project Officer: Jo Ann McCann |