人妻少妇专区

Features

Theophano Mitsa 鈥�91 (PhD) walked into her advisor鈥檚 office one morning in the early 1990s. Mitsa was there to talk about finishing her doctoral dissertation, a highly technical project on something called a 鈥渂lue noise mask,鈥� whose aim was to improve images in printers, fax machines, and computer screens.

鈥淗e said, 鈥榊ou know what, Theo? You have to start thinking about philanthropy,鈥� 鈥� Mitsa recalls. 鈥淚 said to myself, 鈥榃hat is [he] talking about? I鈥檓 hardly scraping by. I need philanthropy.鈥� 鈥�

Sitting across from Mitsa that day was Kevin Parker, a junior professor of electrical and computer engineering at Rochester who had long been intrigued by the intersection of computer technology, medical research, and imaging.

鈥淚 guess he had an inkling this was going to be big from the beginning,鈥� Mitsa says. 鈥淚 never thought that I had anything big. Never in my wildest dreams.鈥�

Judging from that day鈥檚 evidence, Parker, who would go on to become dean of the engineering school for several years, was also adept at fortune telling.

In the midst of Mitsa鈥檚 reams of equations and lines of computer code, Parker saw what would turn out to be one of the most lucrative patents in the University鈥檚 history. The blue noise mask鈥攁 deceptively elegant technological process that transformed how electronic devices in the early days of the digital revolution rendered images鈥攖urned out to be the kind of paradigm-changing discovery that academics live for.

Since that day in the early 1990s, the invention of the blue noise mask has generated some $30 million in royalties for the University, one of the most lucrative in Rochester鈥檚 history. It鈥檚 surpassed only by a Medical Center team鈥檚 work that led to vaccines against human papillomavirus (HPV), now marketed as Gardasil by pharmaceutical giant Merck and as Cervarix by GlaxoSmithKline, and by work by Porter Anderson, professor emeritus of pediatrics, in the lab of David Smith 鈥�58, former chair of the Department of Pediatrics, that led to a vaccine against pneumococcal meningitis, sold as Prevnar by Wyeth (Pfizer).

And while the blue noise mask was invisible to consumers and became a standard part of printer drivers and other software, the story of the invention played out as an often grueling, up-close education in patent law involving some of the biggest players in the computer industry of the 1990s and 2000s.

But as the patents for the technology expire within the next year and as the power of personal computer technology has advanced beyond the often bandwidth-deprived days of the 1990s, the story has shifted to one of legacy for the University and its inventors.

The dollars have allowed the institution to invest in new faculty, programs, and buildings, kick-starting what Parker describes as a 鈥渧irtuous cycle鈥� in which invention begets funding which鈥攐nce reinvested in teaching and research鈥攂egets more invention. That, in turn, brings in more royalties, to the benefit of the institution, its faculty, and students.

鈥淚f you can generate good tech transfer outcomes, that generates new things back at the university,鈥� Parker says. 鈥淭hat allows the teaching and research mission to get bigger and better, and generates another round of things. That鈥檚 the ultimate goal.鈥�

Rob Clark, the dean of the Hajim School of Engineering & Applied Sciences and senior vice president for research, says that, in addition to the financial benefits that have followed from the invention, the blue noise mask illustrates that inventions鈥攅ven ones narrowly focused to solve a specific problem鈥攃an have broad impact. Its success, he says, sends an encouraging message to faculty about the benefit of working to translate basic discoveries beyond the lab.

A Quirk in Human Vision

In many ways, the blue noise mask was the perfect invention at the perfect time. By the early 1990s, desktop computers had a firm hold in homes and offices. The dot matrix printers that accompanied them in earlier years could only print in black and white. But those printers were giving way to ink-jet and laser printers, able to render shades of gray in black-and-white images. They could also create a wide range of colors from just three basic colors using a process called halftoning.

Those early devices, however, were slow, consumed gobs of computing power, and turned out images that awed no one. Among the unimpressed was Parker, a medical imaging expert.

For him, the print quality problems went beyond mere aesthetics. Parker saw CT scans and ultrasound images so poor that there was the chance something important could be missed.

鈥淭he medical images that we鈥檇 put out were awful, really awful, because of halftoning,鈥� Parker says. 鈥淪o I was very interested in that aspect of it. Everyone had an ink-jet or laser printer by then and yet the renditions were terrible.鈥�

Mitsa arrived on Rochester鈥檚 campus as a Fulbright fellow from Greece in fall 1986. Interested in imaging, she nonetheless agreed to do her early graduate work with Edward Titlebaum, a professor of electrical and computer engineering who was working on computer coding related to the Star Wars defense program, Mitsa says.

After receiving her master鈥檚 degree a year later, Mitsa looked at her situation from a practical standpoint. If she received her doctorate based on defense work, her most likely employer would be the U.S. government or a defense contractor. As a foreign citizen, she realized, she was unlikely to get hired for such sensitive work.

So Mitsa talked with Parker, an affable, thoughtful professor and a careful researcher, who was full of ideas. One colleague described him as 鈥渢he kind of guy you hope to find at a university.鈥�

Parker was interested in the problems of quality and speed plaguing halftone images, and set Mitsa to the task of improving them.

鈥淎ll of the halftone techniques of the time left a pattern, what we would call an artifact, a mistake, a characteristic flaw,鈥� Parker says. 鈥淵ou鈥檙e adding a weird pattern onto what should be a natural medical image, whether a CT scan or an ultrasound. If the artifact goes right through a small tumor, you may never see that it鈥檚 a small tumor.鈥�

The process of halftoning creates images by using a limited palette to create an array of tiny dots whose composition and density render different shades and colors. In black-and-white images, for example, the printer prints only black dots, but creates shades of gray by altering the location and density of the dots. Fewer dots and more white space results in lighter shades, allowing the creation of complex images.

Similarly, color images can be made using a halftone process that changes the dot composition and density of just three basic colors: cyan, yellow, and magenta.

Parker learned that the quality problems might be solved by adding 鈥渂lue noise鈥� to the image, a technique that would shift otherwise distracting errors in the printing process to higher wavelengths where, because of a quirk of the human eye, they would be less visible. But the existing blue noise algorithms were slow and cumbersome to use for the computers of that time, and were not free of unwanted patterns. Parker suspected that a fixed array of numbers could be constructed in advance that would contain all the required blue noise patterns. The result, in theory at least, would be images that were more pleasing, more accurate and, potentially, could be rendered more quickly in a 鈥減aint-by-numbers鈥� scheme that any computer could rapidly use.

鈥淏lue noise is high-frequency noise,鈥� Mitsa says. 鈥淵ou want blue noise in halftoning because it takes all the noise in the halftoning process and moves it to the high frequencies where your eye cannot see it. So your eye gets the big picture, the shapes . . . it鈥檚 not disturbed by the distribution of dots in high frequencies.鈥�

In fall 1987, Mitsa set to work, reading everything she could get her hands on about halftoning and then experimenting, making run after run on Rochester鈥檚 mainframe鈥攄esktop computers of the time had nowhere near the required computing power.

鈥淕enerating a good blue noise mask is a really, really tough problem that requires lots of computer time,鈥� Parker says. 鈥淥nce you make a good blue noise mask, once you crack that nut, you have it. It鈥檚 a set of numbers for the computer . . . it can paint-by-numbers your image really quickly and better than other halftone techniques.鈥�

Mitsa input data in the morning, letting programs run all afternoon while she went over the previous day鈥檚 results, and then headed back to the computer lab in the evening to collect each day鈥檚 output.

鈥淭here was just tons of experimentation, tons of it,鈥� she says. 鈥淲e knew we wanted blue noise and, at some point, we tried a lot of different things with blue noise.鈥�

The breakthrough happened one day in late 1988, when Parker walked into Mitsa鈥檚 office brimming with excitement for an idea that he had the night before.

鈥淜evin walked in one morning, very excited, and said, 鈥業 had an idea last night,鈥� 鈥� Mitsa says. 鈥淗e started drawing on the board in my office, and that pretty much was the birth of the blue noise mask because it captured the main idea.鈥�

Neither was sure the idea would work, but eventually, they developed a set of instructions contained in a 256-by-256 grid of numbers that would direct printer drivers and other software how to handle blue noise.

Because the mask was a fixed set of numbers, the computer didn鈥檛 have to constantly recalculate as it created each image. That allowed images to be produced between 5 and 45 times faster than existing techniques. And, at a time when phone companies levied charges on long-distance calls and the office fax machine was an important way to send images and documents, the blue noise mask could cut long-distance fax costs by 75 percent, according to one estimate.

鈥淚f I鈥檓 to describe the blue noise mask, I鈥檇 use two adjectives: it鈥檚 beautiful and it鈥檚 powerful,鈥� Mitsa says. 鈥淚t鈥檚 beautiful because it鈥檚 simple, and it鈥檚 powerful because, although it鈥檚 fixed, although you don鈥檛 have to tweak anything in the mask for any image in the world, the blue noise mask . . . guarantees you鈥檒l get a very high-quality image.鈥�

After developing the first mask, the two continued to tweak it until they had it right. Mitsa also worked on the theory behind the mask, publishing it as part of her dissertation in 1991.

鈥淭o put it in a nutshell, my PhD is actually an array of numbers, which is kind of unique and rare,鈥� Mitsa says. 鈥淚t is a theory, but it鈥檚 also a physical thing, an array of numbers, a fixed array.鈥� In 1991, she and Parker published their first article outlining the work, 鈥淒igital Halftoning Using a Blue Noise Mask,鈥� in SPIE Electronic Imaging Conference (Vol. 1452). In 1992, the U.S. Patent Office issued patent number 5111310, 鈥淢ethod and Apparatus for Halftone Rendering of a Gray Scale Image Using a Blue Noise Mask,鈥� for the invention.

To Market . . . and to Court

Though Mitsa鈥檚 main focus was earning her PhD, Parker understood that the advance had potentially broad commercial applications and began meeting with Research Corporation Technologies (RCT), an outside consultant that, in the early 1990s, many universities used to bring discoveries to market.

RCT Chairman and Chief Executive Officer Gary Munsinger says the advance was so technical that, at first, RCT officials didn鈥檛 understand the value it might have. After the inventors explained it to them, they set to work finding licensees, a process that took a couple of years to bear fruit.

鈥淚t鈥檚 only real when you start getting customers . . . happy customers, repeat customers,鈥� Parker says. 鈥淭hen you really know, OK, this is actually working.鈥�

As encouraging as getting those initial customers was, it became apparent that, instead of paying licensing fees, some players in the industry went ahead and developed their own versions. RCT filed lawsuits against Epson, Lexmark, Hewlett Packard, and Microsoft, two of which went to trial and all of which were eventually settled in RCT鈥檚鈥攁nd the University鈥檚鈥攆avor.

But before settlement, Parker, Mitsa, and RCT鈥檚 legal team had to go through years of court action.

鈥淲e had a saying, that decades come and decades go, because things just took forever,鈥� says Timothy Reckart, then an attorney for RCT and now a partner at Arizona-based Rusing, Lopez & Lizardi.

Though they鈥檙e pleased at the ultimate result of the suits, Reckart says the results weren鈥檛 easily achieved. Part of the challenge, he says, was translating the technology into terms judges could understand and finding experts who could testify.

While the attorneys were in their element, both Parker and Mitsa say the experience was grueling. Parker described their opponents鈥� legal strategy as first denying any violation took place. Losing that, they would argue that the technology they used was different in important ways. Then, when RCT prevailed on those points, they鈥檇 argue that the patent was invalid, sometimes attacking the inventors鈥� integrity to do so.

鈥淚t was just awful, quite frankly,鈥� Parker says. 鈥淭he years of the infringement legal issues were not only very costly and time consuming to all parties involved, but also they were very distressing, very stressful.鈥�

But in the end, also very successful. HP settled with RCT in 1999, taking out a license to use the mask. Epson and Lexmark followed in 2001, the same year RCT filed suit against Microsoft. That case went to trial, with Microsoft prevailing in federal district court in 2005, but RCT winning on appeal in 2008 and 2010.

Whether licensed initially or only after lengthy court battles, the blue noise mask spread widely. Bobby Hunt, a retired University of Arizona engineering professor and imaging expert, says its impact has been extensive. It was in printer drivers and other software, and was important in the operation of the Internet because it cut the amount of data that had to flow over early, slow connections.

鈥淭he impact was immediate in the printer business,鈥� Hunt says. 鈥淚 think virtually all computer printers use blue noise masks or something derivative from that. But there was an impact in the software business, as can be seen in the fact that Microsoft was motivated to embed blue noise masks in the generic printer driver software used for the cross-platform support of printing in all the different printers that may use Windows.

鈥淭here was also impact in the Internet. To make color images requires the transmission of color bytes over a channel. Reducing bits in transmission is important to minimizing user irritation in waiting for an image to load, as well as making more bandwidth available for data shared through the bottleneck of Internet pipes. This is particularly true for images on a cellphone, given the huge growth of smartphones that invite everyone to consume more data.鈥�

Kicking Off the 鈥榁irtuous Cycle鈥�

Blue noise mask licensing ultimately generated millions of dollars in royalties from licensees. After legal fees, the remainder

was split between the University and RCT, with Rochester鈥檚 share further split with the inventors.

Scott Catlin, associate vice president of technology ventures, says the blue noise mask was unusually successful for a faculty invention. The University gets more than 100 disclosures of discoveries from faculty each year, only a few of which ever get to market. He describes the blue noise mask as a once-in-a-decade鈥攐r more鈥攄iscovery.

In an effort to encourage the next blue noise mask, the University is beefing up its technology transfer office, and has created an $800,000 fund to take discoveries through the initial steps of product development, something companies are increasingly reluctant to do, Catlin says.

鈥淭hey [the companies] don鈥檛 want to start from ground zero, they want you to prove that it, at some level, will be a useful product,鈥� Catlin says. 鈥淏lue noise mask is hugely valuable and important. It has produced not just money but also recognition for the University. We鈥檇 love to have more, and that鈥檚 certainly the plan: to maximize opportunities.鈥�

Parker is doing his part, working to ensure the proceeds boost his envisioned 鈥渧irtuous cycle.鈥� Currently the William F. May Professor in Engineering at Rochester, Parker says he became both department chair and dean in part to ensure the proceeds from the blue noise mask were used in ways that would have a positive and enduring impact on research and teaching there.

Blue noise mask revenues endowed five distinguished professorships鈥攖hree in electrical and computer engineering and two in biomedical engineering. Richard Waugh, chair of the Department of Biomedical Engineering, says the money has also allowed the department to hire several junior faculty members, almost doubling the department鈥檚 faculty to 15. During the same period,

biomedical engineering, with more than 300 undergraduate majors, has grown to be the largest undergraduate department in the Hajim School.

鈥淚t came at a critical time,鈥� Waugh says. 鈥淚t enabled us to reach a critical mass of faculty and grow into the department we are now.鈥�

Mark Bocko, Distinguished Professor of Electrical and Computer Engineering, says the blue noise mask funds allowed the department to expand in similar ways, from 15 to 20 faculty members, and also endowed a research initiation fund of about $50,000 per year. That money can be used to help get a project off the ground and running, to the point where it can attract government or foundation funding.

The funds were also used to match a faculty development grant from the Whitaker Foundation, which also awarded Rochester a special grant to help build the Robert B. Goergen Hall for Biomedical Engineering and Optics. The building, which is home to biomedical engineering, was named in recognition of University Trustee Robert Goergen 鈥�60, who committed $10 million for its construction.

Parker himself continued to work on the blue noise mask, guiding graduate students through aspects of research. The license funding also supported work in medical imaging based on biomarkers, which created a spin-off company, Rochester-based VirtualScopics Inc.

While Parker got his virtuous cycle moving, Mitsa just got moving. She took an assistant professorship at the University of Iowa, where she shifted her focus from halftoning to data analysis in medical imaging.

That led to a stint at GE Medical Imaging in Milwaukee and then to Boston, where she was on the faculty of the University of Massachusetts at Dartmouth for a time and a consultant at Harvard Medical School. Along the way, she became interested in a type of data analysis called temporal data mining and wrote the only book in the field. Today, she consults, blogs, and is at work on a second book from her home in Massachusetts.

As the blue noise mask patents expire, the peak of halftoning research has passed, Parker says. While some work continues, the enormous processing power and high resolution of current printers make the speed and quality gains from the blue noise mask less important.

鈥淚n one sense, it鈥檚 like a chess game that lasted 20 years, a very complicated chess game,鈥� Parker says. 鈥淚 couldn鈥檛 be sure what the outcome would be, but I knew if we did everything right, we would be able to make very significant advances. Fortunately, we didn鈥檛 spill the pieces on the way.鈥�

---

Alvin Powell is a freelance writer based in Cambridge, Mass.锘�