Certificate of Approval ILLUMINATING MUSIC: A RESEARCH AND PRODUCT DESIGN STUDY APPLYING SYNESTHESIA AND AMBIENT PERIPHERAL DISPLAY THEORY TO THE VIOLIN Except where reference is made to the work of others, the work described in this thesis is my own or was done in collaboration with my advisory committee. This thesis does not include proprietary or classified information. _____________________________________ Foster Daniel Phillips Certificate of Approval: ________________________ ________________________ Christopher Arnold Tsai Lu Liu, Chair Assistant Professor Assistant Professor Industrial Design Industrial Design ________________________ ________________________ Thaddeus A. Roppel Howard Goldstein Associate Professor Associate Professor Electrical and Computer Engineering Music ________________________ Joe F. Pittman Interim Dean Graduate School ILLUMINATING MUSIC: A RESEARCH AND PRODUCT DESIGN STUDY APPLYING SYNESTHESIA AND AMBIENT PERIPHERAL DISPLAY THEORY TO THE VIOLIN Foster Daniel Phillips A Thesis Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment of the Requirements for the Degree of Master of Industrial Design Auburn, Alabama May 10, 2008 iii ILLUMINATING MUSIC: A RESEARCH AND PRODUCT DESIGN STUDY APPLYING SYNESTHESIA AND AMBIENT PERIPHERAL DISPLAY THEORY TO THE VIOLIN Foster Daniel Phillips Permission is granted to Auburn University to make copies of this thesis at its direction, upon the request of the individuals or institutions and at their expense. The author reserves all publication rights. ____________________________________ Signature of Author May 10, 2008________________________ Date of Graduation iv VITA Foster Daniel Phillips was born on May 15, 1982 to Daniel Ellis Phillips and Donna Foster Phillips in Birmingham, Alabama. He was raised in Homewood, Alabama and graduated as an Honor Graduate from Homewood High School in 2000. He attended Lee University in Cleveland, Tennessee and graduated Magna Cum Laude with a degree in Mathematics in 2004. On August 26, 2006 he married Janet Kathleen Browning. v ILLUMINATING MUSIC: A RESEARCH AND PRODUCT DESIGN STUDY APPLYING SYNESTHESIA AND AMBIENT PERIPHERAL DISPLAY THEORY TO THE VIOLIN Foster Daniel Phillips Master of Industrial Design, May 10, 2008 (B.S., Auburn University, 2005) (B.S., Lee University, 2004) 130 Typed Pages Directed by Tsai Lu Liu See. Hear. Taste. Smell. Touch. The senses are the conduit from the external to the internal. There are a select few who live in an almost unimaginable world where two or more of their senses are joined. There are many varieties of the phenomenon known as synesthesia; one of the most common forms is chromophonia, or colored-hearing. People who have colored-hearing see colors when they hear sound. Often, these colors are directly related to the pitch of the particular notes they are hearing. Is it possible for this concept to be translated for all to experience? vi The process by which a distinctly new instrument, complete with an integrated music illumination system, was created will be documented in this thesis. The steps taken to bring this idea from concept to reality will be thoroughly detailed. These steps include: research into synesthesia, the fundamental concepts of music, ambient peripheral displays, the history of visual music, and the development of the violin and electric violin, as well as user-centered design practices such as the administration of a color-to- tone association survey and physical tests for increased user comfort. The production process of the instrument and the development of the electronic components of the music illumination system will also be documented. In the end, the instrument developed became known as the Vivisi electric violin. The Vivisi electric violin has two main uses: as a performance instrument and a training tool. As a performance instrument the violin uses the music illumination system to create an ambient display of real-time musical rhythm, pattern and dynamics. As a training tool the violin uses the tuning illumination system to provide positive peripheral feedback to those who are learning to play in tune. Both the music illumination system and tuning illumination system have application beyond the electric violin, as does this thesis. The process used to create the Vivisi electric violin can be employed in the design of a variety of products, but it will be most applicable to designs involving the ambient display of information using colored light and to designs of musical instruments. vii ACKNOWLEDGMENTS The author would like to thank Professor Tsai Lu Liu for his encouragement, his wise advice given always at the right moment, and his vast depth of knowledge. Thanks are also due to the other members of the thesis committee, made up of Professor Christopher Arnold, Dr. Thaddeus Roppel, and Dr. Howard Goldstein. Your individual expertise helped to shape this project in a variety of ways. To my parents, Dan and Donna Phillips: Thank you for your endless love and support, it is hard to imagine a life without it. To my wife, Janet: Your complete dedication to seeing this project through has been amazing. You always succeed in your efforts to make my life better. And to all of the other individuals who have affected my life in a positive way, thank you, I appreciate you more than you know. viii Style used: Department of Industrial Design Software used: Microsoft Word, Adobe Photoshop, Microsoft Excel, Rhino, Flamingo, Adobe Acrobat, Apple Safari, Adobe Illustrator ix TABLE OF CONTENTS LIST OF FIGURES ......................................................................................................... xiii 1.0 INTRODUCTION??????????????? ............................................1 1.1 Opportunity Statement?????.????.??????......................1 1.2 Need for Study ??????????.????????... ...................2 1.3 Objectives of Study.........................................................................................2 1.4 Assumptions ??????????????????.?......................3 1.5 Scope and Limit ??????????????????.. ...................4 1.6 Procedure and Methodology ????????????..........................4 1.6.1 Idea Generation................................................................................5 1.6.2 Theory Research ..............................................................................5 1.6.3 Product Research .............................................................................5 1.6.4 Design ..............................................................................................6 1.6.5 Technical Research ..........................................................................6 1.6.6 Survey ..............................................................................................7 1.6.7 Technical Design .............................................................................7 1.6.8 Patenting ..........................................................................................7 1.6.9 Production........................................................................................8 1.7 Anticipated Outcome ????????????????.. ...................8 x 2.0 SYNESTHESIA????????????????..........................................10 2.1 History of Study?????.????????????.......................11 2.2 Prevalence ????????????????????....................12 2.3 Examples??????????????????...??....................14 2.3.1 Edgar Curtis ...................................................................................14 2.3.2 Solomon Shereshevsky ..................................................................15 2.3.3 Michael Watson .............................................................................16 2.3.4 Victoria ..........................................................................................17 2.4 Relevant Studies ????????????????.??.....................18 2.4.1 Ward, etc. Sound - Color Synesthesia ...........................................18 2.4.2 de Thornley Head - Synesthesia ....................................................21 3.0 MUSIC??????????..?????????????..........................24 3.1 Octave and Scale ???????????..??????....................24 3.2 Frequency ???????????..???????...........................26 3.3 Amplitude ???????????..???????......... .................31 3.4 Intonation ???????????..???????.......... .................34 3.5 Musical Expression ???????????..?????....................35 4.0 COLOR SURVEY??????????????????????..............36 4.1 Existing Color Scales ???????????..????.....................36 4.2 Test ???????????..???????.....................................37 4.3 Results ???????????..???????................................40 4.4 Analysis ???????????..???????..............................43 xi 5.0 PERIPHERAL DISPLAYS ????????????????. ......................45 5.1 Terminology ???????????..???????.......................46 5.2 Examples ???????????..???????............................47 5.2.1 Dangling String..............................................................................47 5.2.2 Ambient Orb ..................................................................................48 5.2.3 Speedometer...................................................................................49 5.2.4 Ambient Umbrella .........................................................................51 5.2.5 Swing Setter...................................................................................52 5.3 Application ???????????..???????.........................53 6.0 VISUAL MUSIC PIONEERS??????????????????..... .......54 6.1 Color and Light Organs ???????????..???? .................54 6.1.1 Father Bertrand Castel ...................................................................54 6.1.2 Bainbridge Bishop .........................................................................55 6.1.3 Alexander Wallace Rimington.......................................................57 6.2 Composers ???????????..???????..........................59 6.2.1 Scriabin ..........................................................................................59 6.2.2 Messiaen ........................................................................................61 6.3 Artists............................................................................................................61 6.3.1 Kandinsky and Klee.......................................................................61 7.0 THE VIOLIN ???????????????????????.................65 7.1 The History of the Violin???????..????...???................66 7.2 Construction of the Violin ????????...?????? ...............68 7.3 The History of the Electric Violin ???????????... ...............71 xii 7.4 Instrument Pickups ?????????????????????..75 8.0 CONCLUSION ???????????????????????..............78 8.1 The Idea ?????????????????????.. ................78 8.2 Technical Function of Proposed Solution ????????...................80 8.3 Challenges and Solution of Electronic Components ????... ...............81 8.4 Form Development ?????????????????.. ...............84 8.5 Three-Dimensional Computer Modeling ?????????................89 8.6 Creation of the Model ????????????????. ................91 8.6.1 Non-Electronic...............................................................................91 8.6.2 Electronic .......................................................................................97 8.7 Semi-Working Model ????????????????.. .............100 8.8 Introduction of the Vivisi Electric Violin ????????? .............101 8.8.1 Overview of Results and Benefits...............................................101 8.8.2 Other Applications......................................................................104 8.8.3 Images.........................................................................................105 BIBLIOGRAPHY??????????????????????....................110 xiii LIST OF FIGURES Figure 2.1. Published Papers on Synesthesia from 1850 to 1999.....................................11 Figure 2.2. Synesthetic Family Pedigrees??????????????????.13 Figure 2.3. The Colours Selected by Synesthetes and Non-Synesthetes..........................19 Figure 2.4. The Relationship between Pitch and Lightness..............................................20 Figure 2.5. Local Variance in Same-Pitch RGB Match....................................................22 Figure 2.6. Mean Distance of Quartertone Colour from Set Midpoint.............................22 Figure 3.1. C Major...........................................................................................................25 Figure 3.2. A Minor ..........................................................................................................25 Figure 3.3. Frequency of Instruments in Equal Temperament Tuning.............................27 Figure 3.4. Comparison of the Intervals of the Major Scale.............................................28 Figure 3.5. Notation of the Octaves..................................................................................29 Figure 3.6. Octave Notation and Frequency Tuning.........................................................30 Figure 3.7. Sound Pressure Levels and Corresponding Pressures....................................31 Figure 3.8. Sound Waves in Flux......................................................................................32 Figure 3.9. The Dynamic Ranges of Various Instruments................................................33 Figure 4.1. Three Centuries of Color Scales.....................................................................37 Figure 4.2. Playing a Note in the Flash Based Color/Note Correlation Test....................38 Figure 4.3. Selecting a Color in the Flash Based Color/Note Correlation Test................39 xiv Figure 4.4. A Completed Flash Based Color/Note Correlation Test ................................39 Figure 4.5. Results of the Flash Based Color/Note Correlation Test................................40 Figure 4.6. Most Common Color Choices for Each Note.................................................41 Figure 4.7. How Many People Chose What Note to Pair with Each Color......................42 Figure 5.1. The Dangling String .......................................................................................48 Figure 5.2. Ambient Devices? Ambient Orb.....................................................................49 Figure 5.3. Ambient Devices? Executive Dashboard........................................................50 Figure 5.4. Ambient Devices? Ambient Umbrella............................................................51 Figure 5.5. David Leadbetter?s Swing Setter....................................................................53 Figure 6.1. Bainbridge Bishop?s Color Organ..................................................................56 Figure 6.2. Alexander Wallace Rimington?s Color Organ ...............................................58 Figure 6.3 Performance of Scriabin?s Prometheus, circa 1997 ........................................60 Figure 6.4 Klee?s Variations, circa 1927..........................................................................62 Figure 6.5 Kandinsky?s Fuga, circa 1914.........................................................................63 Figure 6.6 Klee?s Neue Harmonie, circa 1936 .................................................................64 Figure 7.1 Violin (?Charles IX?) by Andrea Amati, Cremona 1564.................................67 Figure 7.2 Violin (?The Messiah?) by Antonio Stradivari, Cremona 1716 ......................68 Figure 7.3 An Exploded View of a Violin........................................................................70 Figure 7.4 National Violectric ..........................................................................................71 Figure 7.5 Rickenbacker Electric Violin ..........................................................................72 Figure 7.6 Rickenbacker Aluminum Tube Electric Violin...............................................73 Figure 7.7 Fender Electric Violin .....................................................................................74 Figure 7.8 Zeta Electric String Instrument Bridge............................................................75 xv Figure 7.9 Barbera Transducers? Twin Hybrid Violin Pickup .........................................76 Figure 8.1 Original Concept ?Napkin? Sketch .................................................................79 Figure 8.2 Strings of the Violin with Colors Applied.......................................................79 Figure 8.3 Screenshot of Tartini .......................................................................................82 Figure 8.4 The Arduino Development Board ...................................................................83 Figure 8.5 Visual Timeline of Electric Violins.................................................................84 Figure 8.6 Electric Violins Ranging from Least to Most Attractive.................................84 Figure 8.7 First Sketch of Developed Form......................................................................85 Figure 8.8 Pink Foam Model of Violin.............................................................................86 Figure 8.9 First Sketch of Developed Scroll.....................................................................87 Figure 8.10 Early Sketch of the Side Profile of the Instrument........................................88 Figure 8.11 Early Sketch of the Neck and Body Interface ...............................................89 Figure 8.12 Screenshot of Acrylic Piece Modeled Using Rhinoceros..............................90 Figure 8.13 Computer Rendering Made Using Rhinoceros with Flamingo Plug-in ........90 Figure 8.14 Computer Rendering Made Using Rhinoceros with Flamingo Plug-in ........91 Figure 8.15 CNC Router...................................................................................................92 Figure 8.16 Foam ?Acrylic? Piece Test Fit with Wooden Top ........................................92 Figure 8.17 CNC Router Making Acrylic Piece...............................................................93 Figure 8.18 CNC Router Making Neck ............................................................................93 Figure 8.19 Wooden Top in Bounding Box from CNC Router........................................94 Figure 8.20 Neck in Bounding Box from CNC Router ....................................................94 Figure 8.21 Body Assembly Test Fit ? Before Sanding ...................................................95 Figure 8.22 Staining..........................................................................................................95 xvi Figure 8.23 Gluing the Violin Together ...........................................................................96 Figure 8.24 Three LED Portion of Electronics Schematic from Lab Chalkboard............97 Figure 8.25 Electrical Lab Work ......................................................................................98 Figure 8.26 Test Fit of Jacks.............................................................................................99 Figure 8.27 First Test of Actual Electronic Components to be Installed..........................99 Figure 8.28 Electronic Components Installed in the Acrylic..........................................100 Figure 8.29 Screenshot of Video ....................................................................................101 Figure 8.30 The Vivisi Electric Violin ...........................................................................106 Figure 8.31 The Vivisi Electric Violin in Concert..........................................................107 Figure 8.32 One-to-One Color-Note Relationship..........................................................108 Figure 8.33 What is the Color of Music?........................................................................109 1 1.0 Introduction 1.1 Opportunity Statement Music has always been an avenue for creative individuals to express themselves. Over time, design and technology have given new and varied paths for musical expression, and have provided more comfort to musicians while they use their creative talents. Although advancements in technology have allowed for the creation of new instruments and have changed how older designs are currently made, many people believe that no instrument maker will ever improve upon Stradivarius? design of the violin (Colwell and Goolsby, 2002). Since many believe that the goal of a perfect violin was achieved in the early 1700s, the design of the classical violin, along with many other musical instrument designs, has seen little change over the last 300 years. The purpose of this study is not to find fault with traditional instruments, but to seek out a new outlet that will allow musicians another way of expressing their creativity. This study is an attempt to find a way that will allow artists to create in two realms of the senses, reaching into both the auditory and visual spectrums. 2 1.2 Need for Study A specific opportunity presents itself for a new development in user-instrument interaction, as well as in user-audience-instrument interaction. The chance to incorporate both the auditory and visual spectrums into one instrument will allow the musical artist to create in a new way that will enhance performances for both the audience the musician. Using the same technology that will be developed for the musical artist, the opportunity is available for those learning to play musical instruments to know when the notes they are playing are perfectly in tune, making practice sessions more productive. These same advancements can bring another opportunity to reality, the chance to give a new understanding to the hearing impaired of the rhythm and subtle volume changes that can be heard in musical performances. 1.3 Objectives of Study In order to create a new instrument with an integrated music visualization system, the following objectives were established. - Research the correlations between music and color that people have, and make a case for combining the two in a single musical instrument. - Conduct a survey to learn more about the relationships people have between distinct notes and color. 3 - Study how passive feedback systems are currently incorporated into product design, and decide how a passive feedback system can best be incorporated into a musical instrument. - Examine the work of composers and artists who claim to have had synesthesia and those who have featured colored light in uncommon situations. - Present a design for a musical instrument that will entertain both the auditory and visual senses. - Create an instrument using the findings from the research and the design process. - Explain the technical functions of the musical instrument. - Provide concluding guidelines for industrial designers working in similar fields or with similar challenges. 1.4 Assumptions At the basis of this thesis study, there are a few vital concepts that are assumed to be true. First, people can associate colors with music. Whether consciously or not, it is assumed that there is an innate association between colors and music that exists in the human brain. Second, an instrument that allows a musician to play colors in conjunction with music will be accepted by the musicians playing it and the audiences seeing the performances. Third, electronic technology is advanced enough to make such an instrument possible. Finally, it is assumed that an instrument that gives positive passive 4 feedback when notes are played in tune will make the process of learning to play in tune easier and more fun. 1.5 Scope and Limit This study is focused on creating a musical instrument. The research and design energy will be focused upon the violin, but with great opportunity for a similar interface and approach with many other instruments. Also, a color study will be conducted in which the pool of subjects will be limited to students, faculty and their associates from Auburn University, Samford University and the University of Montevallo. These three schools are universities in the state of Alabama. The color study will not be limited to musicians, but will include a cross section of many fields of study. In the same manner, the color study will not be limited to subjects with synesthesia and will most likely contain responses from subjects without synesthesia. This will be done in an attempt to understand how the vast population reacts to situations that are commonplace to those who experience synesthesia. 1.6 Procedure and Methodology Creating a new instrument with an integrated music visualization system requires planning and following existing methodology. It also requires determination and creative 5 problem solving, especially when encountering problems that have not been solved before. 1.6.1 Idea Generation After the initial idea, much thought was put into the feasibility of the idea of creating a musical instrument that would allow musicians to create art that integrates the auditory and visual senses. Discussions with a professor of music introduced the topic of synesthesia, which gave direction for the initial research. 1.6.2 Theory Research Case studies and explanations of synesthesia were read and analyzed. Research about synesthesia opened other avenues for research about light organs and composers who used the concept of color in their works. There is a history of composers, artists and scientists who have compared the color spectrum to the musical octave. 1.6.3 Product Research Attempts were made to find products already on the market similar to the proposed musical instrument. After no parallel product was found, the scope of the search widened and several products in different genres were found, pointing to the possibility that the technology and market were both prepared for the proposed musical instrument. Other product research was conducted to examine the ergonomic and aesthetic development of the electric violin. This was done to find a direction for the aesthetics of the proposed product. 6 1.6.4 Design A Gantt Chart was employed to divide the project into smaller, more attainable goals. Rough sketches were drawn to develop an idea of the overall form of the instrument. Once the form had developed significantly, full-size drawings were made so that a more complete understanding of the concept would be possible. The opinions of other industrial designers, professors, and violin players were solicited to solidify the final direction of the overall form of the instrument. Rough foam models were constructed to see how the proposed form would feel and to evaluate the reactions of others. A violin teacher was consulted to find his opinions about the form. Once a significant amount of time had been devoted to the development of the design, a computer model was built to further detail the design. Photo-realistic computer renderings were made so that others would be able to grasp the full idea and give feedback. 1.6.5 Technical Research Several different technical aspects of the project had to be researched in order to insure that the correct parts, when assembled, would synthesize to create a well functioning whole. Electronic pickups for stringed instruments, tuning pegs, chin rests and tailpieces were all studied in an attempt to find the best fit for the new instrument. Also, wood and plastic material types were evaluated for their strength and function. Finally, different types of LEDs (light emitting diodes) were considered, and a particular style was decided upon. 7 1.6.6 Survey A survey was conducted to examine how the average person would interpret a color and musical tone correlation. The hypothesis was made and the study was conducted using a Macromedia Flash based survey which allowed the subjects to match colors with certain tones in an octave. The survey was sent to three college campuses where students, faculty and staff were asked to participate. Over one hundred and fifteen people responded to the survey. The results were tallied and the findings analyzed. 1.6.7 Technical Design An Auburn University electrical engineering professor with a special interest in sensor fusion was contacted, and he agreed to help produce a functioning prototype. After many attempts, a functioning prototype of the electronics was created using a micro-controller, an array of LEDs, a wired connection between the electronics and a computer, and computer software that controlled the array of LEDs. 1.6.8 Patenting During the process of creating the prototype, two professors recommended that the idea be conveyed to Auburn University?s Office of Technology Transfer (OTT). The OTT believed that the idea had enough merit to warrant the completion of a Technology Disclosure Form. The Technology Disclosure Form opened the door for Auburn University to investigate possible routes for provisional patenting in an attempt to market the idea to an existing company. A provisional patent was filed, and licensing of the idea is still a possibility. 8 1.6.9 Production The computer models that were created for the photo-realistic renderings were created in a way that allowed them to be exported to a CNC (Computer Numeric Controlled) Router. There were several attempts to create test pieces for the prototype. The earlier attempts failed because light-weight polyurethane foam was used for the prototype material. Once ren-shape (a denser, more stable foam) and basswood (a wood suitable for carving) were used, the process went more smoothly and test pieces were created to insure that the final prototype could be assembled correctly. An earlier test piece revealed a design flaw that made one of the pieces too flimsy, so structural supports were added in the computer model and another test piece was created. The final pieces were milled out of acrylic, tiger maple, and basswood and required some sanding for their textures to be smooth. Stain was applied to the instrument. The pieces were then assembled along with the electronics, and the prototype instrument was ready. 1.7 Anticipated Outcome The anticipated outcome of this study is four-fold. The first goal of the study is to create a musical instrument that will enhance a musical performance from the points of view of the musician and the audience by involving both the auditory and visual senses. The second goal of the study is to improve students? training by using positive passive feedback to let students know when they are playing in tune. The third goal of the study is to provide a captivating visualization of music for people who are hearing impaired. 9 The fourth goal of the study is to provide a precedent that future designers can use when undertaking similar opportunities and challenges. 10 2.0 Synesthesia Synesthesia can best be described as a melding of the senses. The word comes from the Greek words ?syn? which means together and ?aisthesis? which means perception. Synesthesia can be compared to the word anesthesia, which means ?no sensation.? However, synesthesia means joined sensation (Cytowic, 1996). Stedman?s Concise Medical Dictionary for the Health Professional provides a more technical definition: ?A condition in which a stimulus, in addition to exciting the usual and normally located sensation, gives rise to a subjective sensation of different character or localization; e.g., color hearing, color taste.? Another medical definition can be found in Mosby?s Medical, Nursing, and Allied Health Dictionary: ?a phenomenon in which sensations of two or more modalities accompany one another, as when a visual sensation is experienced when a particular sound is heard.? Synesthetes experience the world in a way that is foreign to most people. Certain sights, sounds, tastes and odors can trigger their other senses causing the world to be quite an interesting place. Most synesthetes learn very early in life that others do not have the same experiences as they do, and most learn to keep their synesthesia a secret for fear of being labeled as strange or different. Later in life though, many have come to realize what a unique gift synesthesia can be and have volunteered for studies that will help us understand more about how the human brain works (Cytowic, 1993). 11 2.1 History of Study The study of synesthesia blossomed in the late 1880s coinciding with the birth of the discipline of psychology. Francis Galton wrote a book published in 1883 entitled Inquiries into Human Faculty. In this book, Galton reveals an interchange that has since been replicated many times when two or more synesthetes discuss their experiences (Harrison, 2001). ?Persons who have colour associations are unsparingly critical. To ordinary individuals one of the these accounts seems just as wild and lunatic as another, but when one seer is submitted to another seer, who is sure to see the colours in a different way, the latter is scandalized and almost angry at the heresy of the former? (Harrison, 2001). Francis Galton is one of the most recognizable names to have delved into the subject of synesthesia in the late 1800s, but there were many others who were interested in the topic. Below is a bar chart that displays the number of published papers on synesthesia from 1859 to 1999. Figure 2.1 Published Papers on Synesthesia from 1850 to 1999 (Harrison, 2001) 12 As can be seen in the previous table, the popularity of synesthesia rose rapidly, but after 40 years, interest waned (Harrison, 2001). The late 1900s saw a resurgence of the popularity of synesthesia, beginning with the work of Richard Cytowic in the 1980s. Research has continued in an effort to understand more about synesthetes and more about the human brain. In February of 2006, the medical journal Cortex released an issue solely devoted to synesthesia. The issue included twenty two articles on the subject of synesthesia. These articles were very detailed and included new studies and research that is relevant to music and color interaction. A few of these articles will be discussed later in detail. 2.2 Prevalence There have been several notable studies conducted to find how common synesthesia is in the modern era. In 1989, Richard Cytowic estimated that 1 in 250,000 people had some form of synesthesia. Cytowic later re-estimated the prevalence was closer to 1 in 10,000 (Harrison, 2001). In contrast, the research team of Baron and Cohen has claimed that the prevalence of synesthesia is 1 in 2,000 (Roberston and Sagiv, 2005). This study was conducted by placing ads in the Cambridge Evening News and the Cambridge University Student magazine, Varsity. The ads requested that those with synesthetic experiences contact the researchers. Twenty-six cases of synesthesia were confirmed out of a total estimated readership of both publications of 55,000 (Harrison, 2001). 13 The Baron and Cohen study also gave evidence that synesthesia is more prevalent in women than in men; the sex ratio combined from the results of both advertisements was 5.5 females for every one male. Synesthesia seems to be passed on generation to generation. No one can be sure whether the link is genetic or is due to surroundings and upbringing. That being said, males rarely pass on their synesthetic traits to sons. Daughters of synesthetes are much more likely to be synesthetic, showing no change if the synesthetic parent is the mother or the father (Harrison, 2001). Figure 2.2 Synesthetic Family Pedigrees from Baron-Cohen et al. (Harrison, 2001) 14 ?Squares represent males, circles females. Horizontal lines indicate the individual was actually tested, rather than the information being derived solely from report. Numbers indicate the individual?s age? (Harrison, 2001). 2.3 Examples 2.3.1 Edgar Curtis In many ways, synesthetes are similar to the average populace. It is unlikely that one could be noticed by watching a busy street or even by engaging in every day conversation. Although synesthetes may seem unremarkable at times, there are occasions when they may reveal how they see the world without even realizing that their perceptions are foreign to a vast majority of the world. Edgar Curtis, son of a Cornell University professor, was discovered to be a synesthete at the age of three and a half. One day, Edgar heard a distant sound of guns being fired at a rifle range. He asked his mother, ?What is that big black noise?? On a different occasion, Edgar questioned his mother about a white noise he heard. His mother responded that it was the sound of crickets, to which Edgar replied, ?not the brown one, but the little white noise?? Edgar Curtis associated colors with many of the sounds he heard in his early years of life, such as the orange sound of whirring electric fans and the black sound of a vacuum cleaner. Edgar also found great delight in sitting alone in a room at a piano, 15 striking a number of piano keys and proclaiming, after each note, its color. ?One day, upon seeing a rainbow, Edgar exclaimed, ?A song! A song!? ? (Dann, 1998) 2.3.2 Solomon Shereshevsky Alexander Romanovitch Luria studied Solomon Shereshevsky?s incredible mind in detail. In 1968, Luria wrote a book about Shereshevsky entitled, The Mind of a Mnemonist. The book is a deep insight into the mind of Shereshevsky, spanning twenty years of study by Luria (Dann, 1998). Solomon Shereshevsky had a seemingly limitless memory and was able to make a living performing as a professional memory expert. Shereshevsky could study a table of fifty unrelated digits for three minutes and then replicate them in forty seconds (Harrison, 2001). His synesthetic experiences were just as unique as his memory. His senses had no boundaries and this helped his memory. To recall a telephone number he had to feel it on the tip of his tongue first. Shereshevsky?s synesthesia allowed him to ?see? each sound or word he heard. The visualization then became an item that he could recall at any time, even years later (Cytowic, 1993). Although Shereshevsky?s memory and synesthesia had some benefits, there were also drawbacks. His synesthesia was so strong that at times it was hard for him to do simple tasks, such as reading, without his synesthetic experiences overpowering reality. Another interesting problem that Shereshevsky struggled with was understanding synonyms. His synesthetic experiences for two similar words could be completely different, so it baffled him that the words could be alike (Dann, 1998). 16 His trouble focusing on one experience at a time is most likely linked to the fact that Shereshevsky had the most intense form of synesthesia that is possible, also known as five-point synesthesia. As a five-point synesthete, Shereshevsky ?experienced secondary sensations in all of his five primary sensory modalities. In addition to color hearing, Shereshevsky had both tactile and gustatory hearing ? he experienced sensations of taste and touch in response to sounds? (Dann, 1998). 2.3.3 Michael Watson Michael Watson is the man who Dr. Cytowic wrote about in his 1993 book, The Man Who Tasted Shapes. As the title of the book suggests, Michael Watson experienced synesthesia related to tastes. When Dr. Cytowic first learned of Watson?s synesthesia, it was because Watson absentmindedly remarked that the chicken he was preparing for Cytowic and other guests did not have enough points on it. Watson, who had recently met Cytowic and knew that he was a neurologist, began to explain his experiences to Cytowic. Watson?s remark about the chicken led to many years of investigation by Dr. Cytowic and eventually gave a clearer picture of one man?s synesthetic experiences. One experiment that Dr. Cytowic conducted on Watson was done in an attempt to find the location where synesthesia occurs in the brain. Watson was subjected to the regional cerebral blood flow technique, also known as CBF. A CBF can give a picture of the brain in action, showing the degree of activity in different parts of the brain during various mental tasks. Data was collected from both Michael Watson?s resting state and while he experienced synesthesia. The results of the tests lead to Dr. Cytowic?s theory 17 that synesthesia is localized in the limbic brain. A standard view of the brain would argue that every mental state is represented in the cortex, so it was quite surprising when Cytowic read the test results from Watson?s CBF and found that Watson?s cortex all but shut down during his synesthetic experiences (Cytowic, 1993). 2.3.4 Victoria Dr. Cytowic also wrote about Victoria in The Man Who Tasted Shapes. She experienced color hearing and occasional color smelling. Cytowic recognized Victoria as a synesthete after his beeper sounded three shrill tones. Victoria responded to the sound by exclaiming, ?Oh, those blinding red daggers! Turn that thing off.? This opened up conversation with her about synesthesia. Unlike Michael Watson, Victoria had heard of synesthesia before and knew that she had it (Cytowic, 1993). Her most common synesthetic responses came from hearing music and presented in the form of splotches of color. Occasionally, her synesthesia presented when she was exposed to environmental sounds, such as the beeper described previously, or mechanical sounds such as clattering machinery (Cytowic, 1989). Victoria?s synesthesia was found to be relative pitch. However, typically she would perceive notes she thought were high in pitch to be pink, and notes she thought were low in pitch to be blue. Rather than having a one-to-one response, like A always being red, her synesthetic experiences were dependent upon the notes that she had heard previously. 18 2.4 Relevant Studies 2.4.1 Ward, etc. Sound-Color Synesthesia Jamie Ward, Brett Huckstep and Elias Tsakanikos of the Department of Psychology at the University College London in London, UK wrote a journal article for the February 2006 issue of the medical journal Cortex entitled ?Sound-Colour Synaesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common To Us All?? The article?s main premise is that there is a common link between how subjects with music-color synesthesia (chromasthesia) and subjects without synesthesia relate the pitch of a note to the lightness of colors. Ward and his colleagues conducted a detailed study of 10 synesthetes and 10 non- synesthetes, playing them 70 different sound stimuli including sets of musical notes from a piano, a string instrument, pure tones, and one set with different instruments playing each note. Two reoccurring themes were noticed when the data was analyzed. First, as a whole, the synesthetes? color matches were more consistent than the non-synesthete group?s color matches after retests of 10 minutes and retests after several weeks. Second, both the synesthetes and the non-synesthetes were similar in their pairing of lower notes with darker colors and higher notes with lighter colors. Figure 2.3 shows the ten colors selected on two different occasions by a synesthete (top) and a non-synesthete (bottom). As can be seen in Figure 2.3, both subjects exhibit a tendency to relate darker colors to lower pitched notes and lighter colors to higher pitched notes. The control subject had a harder time replicating his choices on the retest, but still chose colors that were darker for lower notes and lighter for higher notes. It is also noteworthy to mention the effect of 19 timbre on the subjects. The control subject exhibits no distinction when listening to notes played by the piano, pure tones and notes played on a string instrument, but the synesthetic experience is changed noticeably because of the type of notes. The computer- generated pure tone elicited cooler colors than the more natural instrumental notes. Figure 2.3 ?An example of the colours selected (on 2 occasions) for the 10 single piano, sine (or pure tone) and string notes for a synaesthete (LHM, top) and a control subject (CE, bottom)? (Ward, et al, 2006). 20 Figure 2.4 shows the average responses from the synesthetic group and the control group relating lightness of color to the pitch of notes. Both synesthetes and non- synesthetes show a general tendency to relate higher pitched notes to lighter colors. The synesthetes in the study responded very differently than those from the control groups, this can be seen in Figure 2.3, but the common bond between both groups was the tendency to relate the light colors with high notes and dark colors with low notes. Figure 2.4 ?The relationship between pitch and lightness for synesthetes and controls for the 30 single notes (collapsed across timbres and testing sessions): 0 = darkest, 10 = lightest? (Ward, et al, 2006). 21 2.4.2 de Thornley Head ? Synesthesia Phineas de Thornley Head, a member of the Department of Theoretical and Computational Neuroscience at the University of Plymouth in Plymouth, UK, also wrote an article for the February 2006 issue of Cortex. The article was entitled ?Synaesthesia: Pitch-Colour Isomorphism In Rgb-Space?? The study conducted for the article was specifically geared towards chromasthesia and had five different aims. The first aim was to confirm synesthesia as a verifiable sensory phenomenon. The second aim was to find whether or not the letter names assigned to musical notes affected the synesthetic experience. The third aim was to test quartertone synesthetic experiences to see if colors associated with the pitches in between notes on the chromatic scale would be unique, or if the colors would be between the colors experienced for the notes on either side of the quartertones. The fourth aim was to investigate if synesthetes associate the same hue for the same note at different octaves with different lightness or darkness, or if the hues for the same notes at different octaves would be experienced with no relationship to each other. The fifth aim was to find out if synesthetes have some shared synesthetic experiences, or if all synesthetic experiences are unique. 22 Figure 2.5 ?Local variance in same-pitch RGB match for controls and synaesthetes, no information condition? (de Thornley Head, 2006). Figure 2.6 ?Mean distance of quartertone colour from set midpoint? (de Thornley Head, 2006). 23 Figure 2.5 shows the variance in matching colors to musical notes in both control and synesthetic groups. The synesthetes were found to be able to repeat their matches of colors to specific notes with much greater accuracy. This data helps to solidify the idea that synesthesia is not just an exercise in memory, but a real experience. Figure 2.6 helps to explain the data gathered from the quartertone tests. A quartertone is a note at the midpoint between any two consecutive notes on the chromatic scale. For instance, the quartertone between C# and D would be C#+, or the note half way between C# and D. In this study, synesthetes without absolute pitch, the ability to recognize the letter name of a note simply by hearing it, were able to identify the color associated with that quartertone more often than either the control group or the synesthetes with absolute pitch. Although the synesthetes without absolute pitch tended to match colors more closely, all subjects exhibited proclivity in matching quartertones with their hypothetical color midpoints. 24 3.0 Music Several of the articles mentioned in the previous chapter touched on a few of the finer details of the musical scale. To be able to understand the whole of the studies conducted upon synesthetes and the data that surfaced from these studies, a deeper look must be given to music and its components. Music is defined by the Oxford American Dictionary as ?the art or science of combining vocal or instrumental sounds (or both) to produce beauty of form, harmony, and expression of emotion.? Much of this chapter will focus on the technical and structural side to music. But music, as the definition points out, is more than a structured science; it involves and invokes human emotion, provoking personal expression and interpersonal passions. Music is wholly part of the realms of art and science, drawing from both domains to produce a universal language of sound. 3.1 Scale and Octave ?Any series of tones arranged progressively in intervals of whole-steps or half steps is a scale? (Goetschius, 1934). There are many different varieties of scales, but a scale that contains each note in an octave is known as a chromatic scale. The chromatic scale consists of progressive intervals of half steps. If played on the piano, a chromatic scale would be played using all the keys, white and black (Goetschius, 1934). The notes of the chromatic scale are the notes that make up all of western music (Shanet, 1958); 25 because of the all-encompassing nature of the chromatic scale it is useful when drawing parallels between notes and colors. Also, all other major and minor scales are made from the notes of the chromatic scale. A major or minor scale is made up of eight notes divided by whole or half steps. Major scales and minor scales can be differentiated by the intervals of which they are made. Intervals can be either whole steps or half steps. There are half steps between notes B and C, as well as E and F, in the C major scale. C major is the most common example of a major scale and is made of C, D, E, F, G, A, B, and C an octave higher than the original C. These notes are all white keys on the piano. The intervals that make up the C major scale are as follows: C, whole step, D, whole step, E, half step, F, whole step, G, whole step, A, whole step, B, half step, C. In fact, this interval pattern is what makes a major scale; that the half steps come between the 3 rd and 4 th notes as well as the 7 th and 8 th notes (Shanet 1958). Minor scales have half steps between the 2 nd and 3 rd notes, as well as the 5 th and 6 th notes. Figure 3.1 C Major (Shanet, 1956) Figure 3.2 A Minor (Shanet, 1956) 26 The term octave is used in two different ways. Both have been mentioned, but more explanation may be needed. The Oxford American Dictionary defines octave as: ?a series of eight notes occupying the interval between (and including) two notes, one having twice or half the frequency of vibration of the other. Also, the interval between these two notes, each of the two notes at the extremes of this interval, or these two notes sounding together.? The C major scale, once again, can be used as the example. The group of eight notes that make up the C major scale can be referred to as an octave, but the two C notes, one at the beginning and the other at the end, are said to be an octave apart, and if played together are called an octave. 3.2 Frequency Sound, at its most basic roots, is a wave. Sound waves can be measured to find their frequency by counting waves per second (Olson, 1967). This measurement is referred to as the frequency and is measured in Hertz (Hz). Frequency is directly related to pitch, which is the characteristic of sound that is described by being ?high? or ?low? (Wood, 1975). The human ear is able to hear sounds between approximately 20 and 20,000 hertz (Berg, 1982). The range of the piano, for comparison, is 27.5 Hz to 4186 Hz (Pierce, 1983). The human ear has a comfortable listening range coinciding with the range of the piano (Maconie, 1997). The following chart (Figure 3.3), from Pierce?s Musical Sounds, shows each note on the standard piano keyboard, with the frequency in hertz that is used in the equal temperament system of tuning. The chart also shows the 27 range of many other instruments, including the violin, whose range is from 196 Hz to 2093 Hz. Figure 3.3 Frequency of a Piano in Equal Temperament with Ranges of Other Instruments (Pierce, 1983) The range of any instrument in Hertz is dependent upon the tuning system that is used. It is now standard to tune the note A above middle C on the piano keyboard at 440 Hz. The other notes are tuned according to equal temperament, which, along with the 440 A, has become a standard. The equal tempered system of tuning is a compromise between true intonation and a system where all intervals are equal. In fact, there are no 28 true intervals in equal temperament except for the octave (Wood, 1975). The chart below (Figure 3.4) shows how far the intervals in equal temperament are from true intervals. Figure 3.4 Comparison of the Intervals of the Major Scale with the True Intervals (Wood, 1975) When referring to notes without the aid of the musical staff, it can sometimes become confusing as to which octave the note is a part of. There are high C notes and low C notes, high G notes and low G notes. To reduce the confusion notes are given specific names to correspond with their octave. Although there are several ways to notate octaves, the most common is to follow the letter name with a number starting with 0. The lowest octave is denoted by the 0, rising by one with each octave. The following two figures explain the relationship between the written note on a musical staff and its popular notation. 29 Figure 3.5 Notation of the Octaves (Raichel, 2000) 30 Figure 3.6 Octave Notation and Frequency Tuning of the Notes from C0 to C10 (Raichel, 2000) 31 3.3 Amplitude As frequency is related to pitch, amplitude is related to volume. It has already been stated that sound is a wave. ?The maximum displacement of the medium, during the passage of a wave, from its position of rest is the amplitude of the motion? (Wood, 1975). Amplitude is measured in decibels (db). Very quiet sounds might register at 10 decibels, where very loud sounds will be close to 200 decibels. As can be seen in Figure 3.7, the threshold of pain is 130 decibels. Some frequencies are perceived to be louder than others, but on average most people begin to experience discomfort at 120 decibels and pain at 130 decibels (Raichel, 2000). Figure 3.7 ?Sound Pressure Levels and Corresponding Pressures of Various Sound Sources? (Raichel, 2000) 32 Figure 3.8 Sound Waves in Flux (Berg, 1982) Figure 3.8 contains three examples of sound waves that are changing. These examples properly illustrate changing frequencies and amplitudes and how they relate to what is heard. Example A shows an audible wave that remains constant in pitch, but becomes softer. Example B shows an audible wave that becomes higher in pitch, but remains at the same volume. Example C shows an audible wave that becomes simultaneously softer and lower in pitch (Berg, 1982). Figure 3.9 is significant because it shows the amplitude range of several different instruments, but it is also useful because it illustrates instrumentalists as they try to reproduce ?equally loud? sounds. When attempting to match a previous loudness the musicians studied were able to achieve an accuracy of about 5 decibels. This deviation, paired with the natural quality that some instruments achieve louder tones at higher 33 frequencies, and some instruments achieve louder sounds at lower frequencies, makes playing at a constant loudness up and down the octaves quite a challenge. Figure 3.9 ?The dynamic ranges of various instruments. The broad bands show the relative intensities of the loudest tone that can be produced, as a function of pitch; the lower bands show the relative intensities of the softest tone that can be produced. The width of a band shows the range of intensities that may result when a player tries to produce ?equally loud? tones? (Pierce, 1983). 34 3.4 Intonation There may be no other subject more important to the enjoyment of music than intonation. Notes played out of tune can be anywhere from annoying to almost painful to hear. Some instruments, like the piano, have a fixed tuning. These instruments must be carefully tuned (usually by a professional) before a musician is able to use the instrument to play beautiful music. Other instruments, like the violin, may have a few notes tuned before a performance, but most notes played must be carefully evaluated and sometimes tweaked to be played perfectly in tune. When tuning an instrument to a certain note, musicians may use electronic tuners that display how far away the current note is from the desired notes. These electronic devices use a unit called cents to accurately portray how far or how close a note is to being in tune. There are 1200 cents in one octave (Taylor, 1992). There is no direct correlation from cents to Hertz, but a cent can be understood as a percentage of error. For instance, one cent at the octave from C3 to C4 would equal 0.109 Hz, but one cent at the octave from C5 to C6 is equal to 0.436 Hz, assuming an equal tempered scale. The human ear is able to determine pitch with impressive exactness. Below C5 (524 Hz) the ear is barely able to detect a pitch fluctuation of 3 Hertz. At A1 (55 Hz), 3 Hertz would equal almost one quarter of a whole tone, but at A4 (440 Hz), 3 Hertz would equal only .06 of a whole tone. Above C5 the pitch differentiation threshold is constant at .05 of a whole tone (Winckel, 1967). 35 3.5 Musical Expression Musicians use expression in many forms to enhance a performance. Even though a piece of music may be played from a written score, the personality and emotion of the musician can be heard because of the small nuances that they add while they are performing. These additions are sometimes referred to as ornamentation (Maconie, 1990). Pitch, timing, and continuity are the three categories in which musicians affect the performance (Maconie, 1990). Specifically, trills, slides, heavy vibrato, fermatas, crescendos or decrescendos can be used to spice up a written piece of music. These techniques are often written into a piece of music, but composers must take care that excessive notation does not control the musician to the point where the performance loses the element of spontaneity. Even when spontaneity is a planned event, it is typically still perceived by the audience as unrehearsed, just as it is in masterfully written speeches. Maconie points out that ?orators employ devices of uncertainty and repetition to draw attention to salient points of argument, and to convey an impression of unscripted sincerity? (1990). It is the thrill of the unexpected that makes the experience more memorable and real. 36 4.0 Color Survey Pairing specific colors with single musical notes is not a new idea. Isaac Newton is thought to be the first man to propose the idea (Gerstner, 1986). He paired the seven colors of the rainbow with the seven notes of the natural musical scale. This idea was not rooted in synesthesia, but in the idea that light waves and sound waves should naturally have a direct correlation (Gerstner, 1986). Since then, many others have suggested color scales based upon a variety of theories. 4.1 Existing Color Scales The first known color scale was proposed by Isaac Newton (Callopy, 2004). ?Newton performed his first prism experiments in 1666 and published his theory about the seven colors of the rainbow in his Optiks of 1704? (Brougher, Strick, Wiseman, & Zilcher, 2005). Since Newton?s pairing of the color and sound spectrums, a limited number of color scales were proposed by composers, artists, and inventors. In most cases, those who proposed a color scale came to the idea independently. As can be seen in Figure 4.1, there seems to be somewhat of a trend when starting at C, several color scales transition from red at C to purple at B. There are also plenty of color scales that 37 seem to follow no other format. The color scale of Scriabin is one such scale, and its originality can be attributed to his well-documented synesthesia. Figure 4.1 Three Centuries of Color Scales (Callopy, 2004) 4.2 Test A Macromedia Flash based color/note correlation test was devised to find if there was a common bond between how people relate colors with notes. A computer was used to administer the test and subjects were encouraged to take the test in a quiet environment using headphones. The test allowed subjects to select one of 12 notes from D4 to C#5 played on a violin. The notes were not in order, nor were they labeled as notes. Extreme care was taken to ensure the notes played were exactly in tune, with no vibration and at a 38 constant volume. Subjects were able to play each note as many times as they preferred and to change their answers at any time. The colors chosen for the test were taken from common colors used in color scales proposed in the past. Subjects were assured that there were no wrong answers. Once subjects were satisfied with their color/note combinations, the data was recorded. Figure 4.2 Playing a Note in the Flash Based Color/Note Correlation Test To play a note using the test, subjects used a mouse to click on any number, 1 through 12. The gray box around the number would lighten when the cursor was hovering above the box, and once the box was clicked, the note would sound. Notes could be played at the same time to test musical harmony against color harmony, and could be played in any order or combination. 39 Figure 4.3 Selecting a Color in the Flash Based Color/Note Correlation Test Figure 4.4 A Completed Flash Based Color/Note Correlation Test 40 Colors were selected in a similar manner. When the computer mouse hovered over a colored box, the white border around the box would enlarge. Once the box was clicked it would be marked by the number of the last note played. Subjects then would continue the test until all boxes were marked by an individual number, indicating the note that was selected. The test was administered to 116 subjects. The subjects were adults, aged 18 years and above. No data was collected regarding gender, ethnicity, musical ability or lack thereof. 4.3 Results The most popular answers can be seen in Figure 4.4. D# paired with Maroon was the most popular response with 31% of the subjects identifying that pair rather than any other. Sky blue was the least identifiable color, with nine notes being identified with it somewhere between 7% and 13% of the time. Color to Note Relationship orange D maroon D# khaki green E tan F purple F# blue G sky blue G# yellow A green A# pink B teal C red C# Table 4.5 Results of the Flash Based Color/Note Correlation Test 41 Figure 4.6 Most Common Color Choices for Each Note 42 Figure 4.7 Four Colors and How Many People Chose What Note to Pair with Each Color There was not any significant similarity in the answers of different subjects, but one theme began to emerge. After testing, subjects would seek reassurance that they had answered correctly, and begin to justify their responses. A typical response became, ?I 43 tried to put bright colors with higher pitched sounds and darker colors with the low tones.? 4.4 Analysis A one-to-one color/note correlation is a personal connection made by an individual. Out of 116 participants, no subject created the same color scale. There are 479,001,600 different ways the color test could have been answered, so there is no surprise that there were no repeated answers. The only way there would have been repeated answers would have been if there existed a universal relationship hidden in the depths of the human brain. No subject reacted with extreme confidence in his answers. On the same hand, no one reported the problem of the wrong colors being included in the test. A synesthete would have had trouble with the test because of the fixed colors. The color relationship in a synesthete?s mind is so concrete it would have been extremely frustrating to be forced into answering incorrectly. Although no universal one-to-one color relationship was established from the test, there were some trends that emerged. Similar results were found as in the study headed by Jamie Ward, reported about in Chapter 2. Ward found that synesthetes and non- synesthetes share a common bond of choosing darker colors for lower tones and lighter colors for higher tones. The non-synesthetes from the color test done as part of this thesis work reacted in the same way as the non-synesthetes from Ward?s work. Teal and Yellow were rarely selected for the lower tones in the octave, while maroon and blue were rarely selected for the higher tones. This data leads to the idea that as pitch 44 increases, colors matched to the tone will become brighter, and as pitch decreases, colors matched will become darker. 45 5.0 Peripheral Displays Information is a part of the human experience at all times. The brain is continually processing information, either consciously or sub-consciously. With the advent of computers, the way that people receive and interact with information has changed. Mark Weiser and John Seely Brown have become known as experts in explaining and predicting the course of computing. Weiser and Brown have classified three phases of computing in an overall attempt to interpret how humans will interact with computers in the future. The first phase of computing has been referred to as the mainframe era and was characterized by many people sharing one computer. The second phase of computing, the PC, or personal computer era, began in 1984 when the number of people using personal computers outnumbered those people sharing a computer. A transitional period, with mainframe and personal computing combined by the Internet, will lead us to the third phase of computing. Weiser and Brown have named this third phase the ubiquitous computing era (1997). As the name suggests, ubiquitous computing means computing everywhere, but it also means computing in every thing (Greenfield, 2006). It is best described by the ratio of computers to individuals. In the ubiquitous computing era, there will be many ?computers? for each person (Saffer, 2007). A quoted ?computers? is used to denote the fact that many of the objects in our surroundings may in fact be computers, but will not 46 be what we associate the word computer with as we begin the twenty first century. Ubiquitous computing will be ?invisible? and pervasive (Greenfield, 2006). 5.1 Terminology The ubiquitous computing community has not settled on concrete terminology as of yet. Mark Weiser coined the term ?Ubiquitous Computing,? but Adam Greenfield has chosen to use the term ?Everyware? when referring to the components of ubiquitous computing. Greenfield?s idea of everyware includes a vast number of interconnected microprocessors helping to control environments and to ease the life of those people who are a part of the network. At times ubiquitous computing blends with the background, but when necessary it will move from the periphery to the center of user attention (Weiser and Brown, 1996). This movement between the periphery and the center of attention will be evident in examples such as the Dangling String and the Ambient Orb. Because ubiquitous computing exists mostly in the periphery and users are in control of what enters their center of attention, Weiser oftentimes calls it ?Calm Technology.? The peripheral nature of calm technology leads to displays that reside in the periphery. Peripheral displays are public or personal displays that show information for people to gather when they see fit (Mankoff and Dey, 2003). Peripheral displays can be classified into two groups, ambient displays and alerting displays. Ambient displays present information continuously, oftentimes is an artistic or abstract way, and do not attempt to move into the foreground of the user?s attention. In contrast, alerting displays show information at discrete intervals and do attempt to gain the attention of users at 47 times when notification is necessary (Mankoff and Dey, 2003). Devices that have output a user has no control over are labeled as displays, but devices whose output can be changed by a user?s actions are labeled as interfaces (Mankoff and Dey, 2003). Peripheral interfaces certainly are a growing movement, but will remain outside the scope of this particular study. 5.2 Examples 5.2.1 Dangling String The original and most cited example of a peripheral display is the dangling string. Created by artist Natalie Jeremijenko, the "Dangling String" is an 8 foot piece of plastic spaghetti that hangs from a small electric motor mounted in the ceiling. The motor is electrically connected to a nearby Ethernet cable, so that each bit of information that goes past causes a tiny twitch of the motor. A very busy network causes a madly whirling string with a characteristic noise; a quiet network causes only a small twitch every few seconds. Placed in an unused corner of a hallway, the long string is visible and audible from many offices without being obtrusive. (Weiser and Brown, 1996) 48 Figure 5.1 The Dangling String (Weiser and Brown, 1997) The dangling string creates a window into the network (Weiser and Brown, 1997). Information that was before invisible now has a visible outlet. When the string was first installed it was a center of attention because of its newness and novelty, but after the initial unveiling, the string became part of the background. It became a peripheral display of the network connection traffic (Weiser and Brown, 1997). 5.2.2 Ambient Orb Ambient Devices, a company based in Cambridge, Massachusetts started by graduates of MIT?s Media Lab, released their first product, the Ambient Orb, in 2002. The Ambient Orb is able to display data from a variety of sources, but its main purpose 49 Figure 5.2 Ambient Devices? Ambient Orb (Ambient Devices, 2007) was to display real time stock market trends. Ambient Devices has a network of information that is broadcast to the individual products that it sells. Customers may tap into data about the stock market, weather, pollen forecast, wind speed, traffic congestion, and sports teams (Ambient Devices, 2007). Once the connection is set up, data that wirelessly enters the orb controls its color, giving customers the ability to glance at the object and gather the information they desire. 5.2.3 Speedometer The idea of a peripheral display may seem like a strange and new idea, but the concept and application has preceded the computer revolution. Many people use a peripheral display on a daily basis in their cars. The speedometer is a device used to 50 monitory the speed at which the car, as a result of input from the driver, is traveling. The speedometer is a passive object, moving in smooth increments in a non-distracting way. When the driver needs information relating to the speed of the car, he is able to glance at the speedometer and in an extremely short amount of time receive the information that he needs. Ambient Devices has taken the concept of the speedometer and merged it with its national data network. The ?Executive Dashboard? made information typically accessed by using a computer available in glanceable form, ranging from air quality or pollen count to weather conditions. Figure 5.3 Ambient Devices? Executive Dashboard (Ambient Devices, 2007) 51 5.2.4 Ambient Umbrella Another notable product created by Ambient Devices is the Ambient Umbrella. The umbrella moves the ambient data display out of an abstract device into an object that is familiar. The handle of the Ambient Umbrella contains sensors that wirelessly connect Figure 5.4 Ambient Devices? Ambient Umbrella (Ambient Devices, 2007) to a source providing weather data. When there is a chance of rain, lights in the handle glow blue (Saffer, 2007). The pattern of the lights changes according to what type of precipitation is forecast, whether it is rain, drizzle, snow or thunderstorms (Ambient Devices, 2007). 52 5.2.5 Swing Setter Professional golfer and swing coach, David Leadbetter has lent his name to a product designed to improve a golfer?s swing. Although this product does not have an ambient display, one of its unique features qualifies as a peripheral interface. The Swing Setter is a tool used by golfers who wish to improve their golf swing. When the Swing Setter is used properly, there are two clicking noises you'll hear at specific points in the swing. There is a click just before halfway in the backswing to ensure wrist cock and a click at the point of impact to ensure proper release. Leadbetter says: The Swing Setter's Adjustable Magnetic Balls (settings vary depending on strength and swing speed) promote a correct setting of the wrists on the backswing - setting the stage for an accelerated release through impact? The Swing Setter provides a visual, audible, and tactile response the moment of clubhead release. This translates to more consistency and a greater connection to the point of impact. (Smith, 2005) The adjustable magnetic balls should be considered a peripheral interface. The noise made does not distract a golfer from his original intent, the golf swing, but instead gives positive feedback for a correct action. 53 Figure 5.5 David Leadbetter?s Swing Setter (Smith, 2005) 5.3 Application Peripheral displays should be ?glanceable? and non-interruptive (Matthews, 2007). The main point of a peripheral display is that it does not demand attention. It is acceptable if an ambient device goes unnoticed, for it will be ready in the background when information is needed (Saffer, 2007). ?In practice, most displays will be interruptive initially, but a goal for peripheral display designers is to create a display that lends itself to operationalization? (Matthews, 2007). In using the term operationalization, Matthews brings to light the idea that peripheral displays should be easy to use and easy to learn to use. It will take a shorter amount of time for users to learn how to use peripheral displays and interfaces that have been designed to enable ?quick and easy visual information intake? (Matthews, 2007). 54 6.0 Visual Music Pioneers Since Newton?s assumption that light waves and sound waves should have direct correlations to one another, there has been a spattering of composers, scientists and artists who have worked with color and sound. Spanning different languages and cultures, these forerunners of visual music, often working independently, came to acknowledge the beauty and power that could be attained by combining the two media. 6.1 Light and Color Organs 6.1.1 Father Louis-Bertrand Castel Louis-Bertrand Castel was a mathematician and Jesuit priest who lived in France in the 1700s (Callopy, 2004). In 1725 Castel wrote the first of two essays outlining his idea of a clavecin oculaire or a harpsichord for the eyes. Later, in 1735, Castel?s more developed ideas appeared in M?moires de Tr?voux, under the title ?Nouvelles experiences d?optique et d?acoustique? (Peacock, 1988). It is thought that Castel originally had no plans to actually build the instrument that he had envisioned, but when his ideas were received with skepticism, he set himself to create a model of the clavecin oculaire. Castel never finished the model, or at least there is no permanent record of the 55 instrument (Peacock, 1988), but his idea of a color-music instrument was hotly debated by many of the preeminent minds of the time (Hankins and Silverman, 1995). In general, the reception to the idea of the instrument was not favorable, but it was discussed enough that Castel questioned his critics about why they would spend so much time and energy denouncing what they claimed to be a worthless idea. Voltaire did much to bring Castel?s ideas to a broader audience when he wrote about Newton?s color-tone analogy along with Castel?s instrument in Chapter 14 of his book, El?ments de la philosophie de Newton (Hankins and Silverman, 1995). Although Castel never completed a working color organ, his ideas were carried forth several times in the next century. 6.1.2 Bainbridge Bishop Bainbridge Bishop built a functioning full size color organ in the year 1877 (Callopy, 2004). Bishop resided in New Russia, New York and wrote about his color organ in A Souvenir of the Color Organ, with Some Suggestions in Regard to the Soul of the Rainbow and the Harmony of Light. Bishop writes about the history of ?the analogy which is thought to exist between music and color? and states that no practical color- instrument had yet to be built. Bishop?s most successful color organ, in his own opinion, (seen in Figure 6.1) had large ground glass about five feet in diameter, framed like a picture, and set in the upper part of the instrument. On this the colors were shown. The instrument had little windows 56 glazed with different-colored glass, each window with a shutter, and so arranged that by pressing the keys of the organ the shutter was thrown back, letting in colored light. Figure 6.1 Bainbridge Bishop?s Color Organ (Bishop, 1893) Bishop?s instrument was capable of displaying chords, with the bass notes shown spreading across the lower portion of the ground glass, and the other colors were shown above with their edges blending into one another. Bishop did not think that a person could recognize a song by seeing the colors alone, although the color organ could be played with or without sound courtesy of a stop, but he did believe that a person familiar 57 with the instrument might recognize some melodies (Bishop, 1893). Bishop?s instruments attracted some attention, in fact, P.T. Barnum, the famous showman, had one of Bishop?s color organs installed in his home in Connecticut (Peacock, 1988). It seems that almost no permanent record of Bishop?s work remains. Bishop states that he believes ?there is not a shred of all my five years? work in existence, as I have been told the organ I exhibited in New York was burned with P.T. Barnum?s country house at Bridgeport. One burned at a hotel on Lake George; the other was burned in my own house? (Bishop, 1893). Although Bishop?s work may not still be intact, his efforts and creative spirit are still remembered. 6.1.3 Rimington In 1893 Alexander Wallace Rimington patented his version of the color organ (Callopy, 2004). It is thought that Rimington was the first to use the name ?colour- organ? and since his time the term has been used to describe other similar instruments (Peacock, 1988). Rimington, a professor of fine arts at Queen?s College in London, received more notoriety for his color organ than did Bishop (Peacock, 1988). In 1895, Rimington debuted his color organ at St. James Hall in London (Hankins and Silverman, 1995). Another showing held as a private lecture-demonstration the same year was attended by over 1,000 people. The color-organ, which did not produce sound, was accompanied by a piano, and organ and an orchestra (Peacock, 1988). 58 Figure 6.2 Alexander Wallace Rimington?s Color Organ (Peacock, 1988.) Rimington?s color organ was over 10 feet tall and was powered by a power supply that provided the instrument with 150 amps. The keyboard, similar to that of a standard organ, spanned five octaves. The keys were connected to a series of trackers to a matching set of diaphragms in front of special lenses. Light came from 14 arc lamps and shown through filters that were varnished with aniline dye. ?Stops were furnished to control perception: hue, luminosity, and chroma (color purity). One stop allowed the 59 performer to spread the spectrum band over the entire keyboard instead of over on octave? (Peacock, 1988). Colour-Music: The Art of Mobile Colour was published in 1911. The book was Rimington?s explanation of the theory behind the color organ, as well as a description of the instrument. In the book, Rimington states his conviction that color and sound were connected through some sort of physical analogy, but he acknowledged that the analogy was a broad one and that his theory was open to question. 6.2 Composers 6.2.1 Scriabin The Russian born composer, Alexander Scriabin was the first composer to include a part written for projected light. (Peacock, 1988). Prometheus, the Poem of Fire, Scriabin?s famous 1911 color-symphony, called for the use of a color-organ, similar to that of Rimington?s. It is likely that Scriabin had heard of Rimington?s work, but a different instrument was used for the production of Prometheus. Reviews of the opening night state that the part composed for projected light was missing, due to a malfunction of the equipment. The first public performance that included the projected light was in Carnegie Hall on March 20, 1915 (Peacock, 1988). The part for the color-organ illuminated the fundamental pitch of the composition, usually following the bass line of the piece (Hull, 1918). Scriabin was a synesthete. His synesthetic experiences spanned three senses: hearing, sight and smell (Brougher, et. al, 2005). Scriabin planned to combine all of 60 these senses in a symphony he never finished. Known as Mysterium, the performance would last for seven days and at the culmination of the piece would be the ?cessation of time? (Brougher, et. al, 2005) and a new race of men would be born (Bowers, 1973). As can be assumed from his predictions for the performance of Mysterium, Scriabin, who is considered to have been a Theosophist, had ideas about theology that were not shared by many others. Serge Koussevitzky, Scriabin?s publisher, is quoted as saying that at the end of the Mysterium, ?We will all go out and have a fine dinner? (Bowers, 1973). Figure 6.3 Performance of Scriabin?s Prometheus, circa 1997 (Brouger, 2005) 61 6.2.2 Messiaen Although Messiaen never wrote a part for projected light like Scriabin, he is another composer whose synesthetic experiences helped to shape his music. One anecdote that illustrates the power of Messiaen?s color-music synesthesia takes place when he went to a theater production that paired the music of Beethoven with violet stage lighting. ?The color violet and the key of G major produce an absolutely appalling dissonance,? Messiaen is quoted as saying, ?it clashed terribly and made me sick to my stomach? (Brougher, et. al, 2005). Messiaen?s synesthesia manifested itself in works such as Chronochromie (1959- 60) and Couluers de la cit? celeste (1963). Messiaen spoke about his synesthetic experiences, saying, ?for me, certain sonorities are linked to certain complexes of colors, and I use them like colors, juxtaposing them and putting together in relief one against the other, as a painter enhances one color with its complement.? (Brouger, et. al, 2005). 6.3 Artists 6.3.1 Kandinsky and Klee The lives and Wassily Kandinsky and Paul Klee intersected several times. From 1911 to 1914 Kandinsky and Klee were members of the Blaue Reiter circle, and in the 1920s and early 1930s the two men taught at the Bauhaus. Although they had different styles, large amounts of their works dealt with the visualization of music and synesthetic experiences (Tower, 1978). 62 Klee, a Swiss painter, was a great lover of music. He had a strong love of eighteenth-century classical music and was a violinist. In fact, he trained and performed musically with the Bern Music Society before he turned his main focus to visual art. Between 1920 and 1936, Klee honed his ?analogies between musical counterpoint and color gradation and between formal sequences and compositional arrangements in painting? (Brouger, et. al, 2005). Klee focused on the translation of the beauty of Figure 6.4 Klee?s Variations, circa 1927 (Brouger, 2005) 63 polyphony into paintings. It was an intensely mathematical process. Kandinsky?s works, in contrast, typically began with atonal music and were guided much more by emotion (Brouger, et. al, 2005). Wassily Kandinsky was born in Moscow, but spent many years of his career in Germany. From 1909 to 1914, ?he developed an entire aesthetic theory and a large group Figure 6.5 Kandinsky?s Fuga, circa 1914 (Brouger, 2005) 64 of abstractions conceived in musical metaphor? (Brouger, et. al, 2005). Kandinsky, also a violinist, conceived a three step process by which artist could take a musical composition and abstract it into a visual work of art. The first step was Impressions, which was embodied by a quick sketch done to capture the ?artist?s immediate visual sensation of external reality? (Brouger, et. al, 2005). Next, Impressions were converted to Improvisations. Then the artists would expand Improvisations into a Composition where the process reached its highest level of creativity and free expression (Brouger, et. al, 2005). Figure 6.6 Klee?s Neue Harmonie, circa 1936 (Brouger, 2005) 65 7.0 The Violin The violin attracts musicians and listeners the world over. It has been a force in the musical world since its origins in the 1500s. H.R. Haweis, author of Old Violins and Violin Lore, gives a spectacular explanation of the lure of the violin. What is the secret of the violin? Why is it that when a great violinist appears all the other soloists have to take a back seat? The answer is: the fascination of the violin is the fascination of the soul unveiled. No instrument ? the human voice hardly excepted ? provides such a rare vehicle for the emotions ? is in such close touch with the molecular vibrations of thought and with the psychic waves of feeling. But whilst the violin equals the voice in sensibility and expression, it far transcends it in compass, variety and durability. Consider the singular completeness and perfection of this instrument as a sort of physical and vibratory counterpart of the soul. The four strings no doubt limit and define its compass, and only in the quartet and collectively, 66 is it capable of extended effects of complex harmony; but as a tone producing instrument and within its limits it is perfect ? every gradation of sound between tone and semitone is attainable, and for no other instrument can this be claimed. (Haweis, 1898) 7.1 The History of the Violin Mystery clouds exactly when and where the violin became the instrument that we know today in the modern era. Most violin historians point to Andrea Amati as the violin maker who solidified the shape and size that is now standard. The oldest surviving violin in existence, was made in 1564 by Andrea Amati, in the Italian city of Cremona. Known worldwide for it famous violin makers, Cremona was the home of the Amati family, the Guarneri family and home of Antonio Stradivari (Stowell, 1992). The skilled work of Andrea Amati rose to become the first standard of violin making. Amati defined the shape of the instrument as well as the ?classical form of the scroll and the single row of purfling around the gently channeled edges of the body? (Stowell, 1992). By 1710, the design of the violin was set, mostly due to the advances made by Antonio Stradivari. Stradivari ?reduced the height of the table and back, thereby producing a more powerful sound (Stowell, 1992). He also developed a new system of thicknessing, created straighter and stronger ?C? bouts, and introduced a strong red pigment to the varnish. Stradivari?s ?Golden Period? ranged from 1700 to 1720, in which 67 he produced many masterpieces. The best of these instruments are the ?Greffuhle,? the ?Ernst? and the ?Viotti? all made in 1709 (Farga, 1969). In 1716, Stradivari crafted a violin known as the ?Messiah? violin, now one of the most celebrated violins in the world due to its quality and highly preserved condition (Stowell, 1992). Figure 7.1 Violin (?Charles IX?) by Andrea Amati, Cremona 1564 (Gill, 1984) 68 Figure 7.2 Violin (?The Messiah) by Antonio Stradivari, Cremona 1716 (Gill, 1984) 7.2 Construction of the Violin A typical violin is made of many different pieces and several types of wood. The wood is chosen for its strength, weight and resonance. The table, or front, of the violin is made from two matched pieces of spruce seamed so that the grain of the two pieces will be the mirror image of one another. The table is carved to a finished thickness of 3 mm (Stowell, 1992). The back of the instrument is typically made of two matched pieces of maple, but it is not uncommon to find instruments with backs made from one solid piece of maple. The thickness of the back ranges from 5mm in the center of the instrument to nearly 2.5 69 mm at the edges. The six curved ribs, made of maple, join the table and back, with help from small strips of pine or willow to ensure a proper glue joint (Stowell, 1992). The neck, pegbox and scroll are all carved from one piece of maple, and then an ebony fingerboard is glued in place. Other ebony pieces include the nut, saddle, end- button, and tailpiece. Boxwood or rosewood is sometimes substituted for ebony and is often used for the pegs (Stowell, 1992). One important aspect of the violin making process is the addition of the purfling. Inlaid near the edge of the table, the purfling is made of two outer strips of pearwood and one center strip of poplar, the entire assembly being less than 1.5 mm thick. Not only does the purfling have a purpose as a decorative element, it also prohibits cracks formed on the outer edge of the table from spreading further (Stowell, 1992). The bridge supports the strings and is made from a sliver of maple, carved to allow the vibrations of the strings to be transmitted into the table of the violin. The top of the bridge is curved to allow each string to be played one at a time. On the interior of the instrument the sound post and bass bar support the pressure that is applied to the bridge by the string tension. The sound post is a small rod of spruce or pine, wedged vertically between the table and the back under the treble foot of the bridge. The bass bar is also made from spruce of pine and is glued in place under the bass foot of the bridge, running almost from end to end inside the violin. The bass bar provides needed support and helps to disperse the vibrations of the lower strings throughout the instrument (Stowell, 1992). All aspects of the construction of the instrument affect its tone quality. The violin is held together by animal glue because it can be re-dissolved if repairs are necessary. Long thought to be the secret of great tone, the varnish is applied to enhance the beauty 70 of the wood, and to protect it, but the make-up of the varnish will give different varieties of flexibility and hardness. A poor varnish can ruin the tone of a well made violin, while a good varnish will allow the violin to be played and enjoyed for hundreds of years (Stowell, 1992). Figure 7.3 An Exploded View of a Violin (Stowell, 1992) 71 7.3 The History of the Electric Violin The first electric violins began appearing in the 1930s, in both familiar and unusual shapes. In 1938, National ran an advertisement for ?a highly perfected electric violin? called the Violectric (Schussler, 2001). The Violectric was modeled after the traditional shape and form of the violin. The only sign of its amplified nature was the addition of a volume knob and pickup. Figure 7.4 National Violectric (Shussler, 2001) 72 The same year, the musical instrument company Rickenbacker filed a patent for a very different electric violin (Schussler, 2001). It had no body, no scroll, no peg box and no tailpiece. The pegs were mounted near the chin rest and the strings were strung backwards. This instrument relied solely on a bridge-mounted pickup for amplification of sound ? no resonance was provided by the instrument (Beauchamp, 1938). Figure 7.5 Rickenbacker Electric Violin (Beauchamp, 1938) 73 A few years later, Rickenbacker filed a patent for another distinctive electric violin. Built around an aluminum tube, the electric violin was recognizable as an amplified instrument from quite a distance. The aluminum tube violin had a slightly more traditional look that the previous Rickenbacker violin. Its pegs and a modified scroll were at the end of the instrument, but the instrument was still quite different from any violin ever made before (Beauchamp, 1943). Figure 7.6 Rickenbacker Aluminum Tube Electric Violin (Beauchamp, 1943) 74 Electric violins were rare even in the 1950s when the electric guitar gained popularity. Leo Fender, known most for his electric guitar designs, filed a patent for an electric violin in 1958. His design was not as radical as the Rickenbacker design, but neither did it resemble a traditional violin. One design feature to note in the Fender electric violin is the guitar-like head in place of a traditional violin scroll. (Fender, 1961). Figure 7.7 Fender Electric Violin (Shussler, 2001) 75 Zeta violins reached the market in the 1980s (Mathews, 1989). Backed with a proprietary pickup developed at Stanford University that gave the violins the ability to be MIDI controllers, the Zetas quickly became the standard for many professional electric violinists. Figure 7.8 Zeta Electric String Instrument Bridge (Mathews, 1989) 7.4 Instrument Pickups A pickup is the device that transmits sound energy from an instrument to an amplifier. It is a useful device when volume is an issue, or when certain sound effects are desired. A pickup uses a transducer to convert the sound from the instrument into a form that will be easily used in amplifying the sound. There are two main types of 76 transducers: piezos and magnetic coils. Typically, piezos are more likely to be used in the amplification of acoustic instruments, such as violins, and magnetic coils are more likely to be found in electric guitars (Schussler, 1999). Transducers are small crystals that respond to even the faintest vibrations and changes in air pressure. They are much more responsive than magnetic coils, and will amplify the finer points of the instrument. This is desirable for violins, but piezos can also be so sensitive to the vibrations of the instrument body that they are in danger of producing feedback, especially when used in loud environments (Schussler, 1999). Barbera is a maker of piezo transducers for the amplification of string instruments. Barbera?s violin pickups come in a form that is thicker than, but otherwise similar to, a traditional violin bridge. Figure 7.9 Barbera Transducers? Twin Hybrid Violin Pickup (Barbera, 2007) 77 A pickup allows an electric violin to be unbounded by the rigid acoustic standards that were developed long ago by famous violin makers like Amati and Stradivari. This is not to say that the make-up of the instrument does not matter when a pickup is used, but it does mean that a much wider variety of shapes, sizes and materials are available and when using these varieties a wonderful tone is still attainable. 78 8.0 Vivisi 8.1 The Idea The concept for a system of illuminating the music of a stringed instrument directly in the body of the instrument began to emerge while evaluating data taken from the color surveys, and while conducting research into synesthesia, the fundamentals of musical theory, peripheral displays, and investigating the work of the pioneers of visual music. It was decided that a violin would be a good choice as the instrument for the implementation of the music illumination system because of its nature as a solo instrument, typically playing single notes. Whether by itself on the stage, or backed by a full orchestra during a concerto, the violin stands out. Also, it is possible to play any pitch between 196 Hz to ? 2100 Hz because the violin, along with other members of the string family, is a non-fretted instrument. The violin is also the smallest member of the string family, making production a manageable task. The first idea was to create an instrument entirely out of acrylic, with seven individual chambers. Each chamber, except for the center chamber, was arc shaped, increasing in size to visually mimic sound waves leaving the center of the instrument. Each chamber would be triggered by different volume intensities, with the lowest volumes only activating the center chamber, and the loudest volumes activating all seven chambers, colors being created by the different pitches played on the instrument. 79 Figure 8.1 Original Concept ?Napkin? Sketch This first idea was a start, but it was determined that more exciting, and possibly more ergonomic, solutions were still waiting to be discovered. As the form developed, so did the illumination system. The system pairs each playable note with a specific color, which would allow an observer to see the rhythm and pattern of music played on the instrument. Volume controls the overall brightness of the light, so that the dynamics of the music can be seen. Using the data from the color survey, the mid-ranges of the violin were matched with the most common answers from the survey. Since no two color scales were the same, it was determined that no universal color scale existed, so majority was the determining factor of the color scale used in the illumination system. Figure 8.2 Strings of the Violin with Colors Applied 80 The color survey did uncover one trend in color scales. Generally speaking, darker colors were associated with lower tones, while lighter colors were associated with higher tones (Please see Figure 4.7). This information was used when determining what colors would be used for the different octaves of the violin. The octave starting at D4, or the D string would be the true colors determined by the color survey. The octave below D4 would use shades of the mid-range octave, while the octaves above D4 would use tints of the mid-range octave, eventually turning to white around 1500 Hz at G6. For instance, G3 is a dark blue, G4 is blue, G5 is a brighter, whiter blue and G6 is white with a hint of blue. At this point the idea of an instrument containing an integrated music illumination system had been solidified, but what the instrument would look like or how it would function was still a mystery. 8.2 Technical Function of Proposed Solution The help of Dr. Thaddeus Roppel, associate professor in the Auburn University College of Electrical and Computer Engineering, was solicited to help bring the idea of a music illumination system to reality. The idea seemed simple at first. The audio signal coming from the violin would need to be analyzed for its frequency and amplitude, and, depending on real-time values, a microcontroller would determine which lights should shine and what each light?s color and brightness should be. LEDs are a good choice for the lighting elements because of their small size and low heat output, plus their ability to easily create multiple colors. Full color, or RGB, LEDs have three different colors 81 inside one translucent bulb. By controlling each color separately, a multitude of color variations are available by mixing red, green and blue. Pulse Width Modulation, or PWM, is the process used to control the brightness of each individual color. PWM is a way to turn the LEDs on and off so quickly the human eye is unable to detect a flicker. Brightness is then controlled by how much time the LED spends emitting light. The microcontroller would work very similarly to an electronic tuner, but would have a very different output. Rather than a display of letters or numbers, the output would simply be an array of LEDs located on the interior of the instrument. Unlike electronic tuners, the microcontroller would also need to respond to the amplitude of the sound wave. The illumination system would be powered by an onboard battery pack, and a wireless transmitter could be used to transmit sound data to an amplifier, so it is possible that no external wires would be necessary in the instrument. Because of the complexity of the internal system, some way of accessing the electronic components would need to be incorporated into the design of the violin. 8.3 Challenges and Solution of Electronic Components The main problem to creating a workable solution for the music illumination system was the real-time analysis of the audio signal to find the pitch. Pitch detection is an intensive process. Investigation revealed that pitch detection is a multi-step calculation beginning with a Fast Fourier Transformation, or FFT, of the signal. The FFT is used to find the fundamental frequency of a sound wave, but when conducted on sound 82 created by a bowed stringed instrument, overtones and harmonics were oftentimes so powerful that they fouled the output of the FFT. Electronic tuners then run a series of calculations that rule out the harmonics and overtones to accurately calculate an exact pitch. The complexities of these calculations made it seem unlikely that a real-time solution would be possible. A major breakthrough occurred when the doctoral work of Philip McLeod surfaced in an Internet search related to pitch detection algorithms. McLeod developed a computer program called Tartini, designed to aid vocalist and solo instruments while in practice sessions. Tartini is able to show musicians several different real-time outputs, including tuning accuracy, dynamic patterns, and constancy of vibrato. One feature of the program that lent itself to the application of the music illumination system was an analysis of a recorded sound file. The analysis is able to accurately display the frequency and amplitude of the sound wave every 46 milliseconds (McLeod, 2007). Figure 8.3 Screenshot of Tartini 83 Almost simultaneously with the discovery of McLeod?s work, was the breakthrough of finding the Arduino development platform. The Arduino is a microcontroller with open-source software and a large and active web community (Arduino, 2007). Once the basics of the Arduino programming language were learned, the Tartini analysis was put to use. By exporting the Tartini analysis to Excel, simplification and translation into the Arduino programming language was able to be made. Unfortunately, the Arduino has a very limited amount of on-board memory, so oversimplification of the Tartini was necessary, virtually erasing all data regarding amplitude. Even so, a realistic depiction of the music illumination systems was able to be realized by video. Figure 8.4 The Arduino Development Board (Arduino, 2007) 84 8.4 Form Development The shape of a traditional violin has been well developed for acoustic and visual appeal, but an addition of an electronic pickup allowed exploration of a new form. The new form to be developed began to take shape in sketches, over a few months. Common shapes for electric violins in the past were studied to get a feel for their visual successes and trends that might lead to a current instrument. Current electric violins were also studied to understand what instruments were visually successful and also exuded a sophisticated aura. Figure 8.5 Visual Timeline of Electric Violins Figure 8.6 Electric Violins Ranging from Least to Most Attractive Sketching became the tool of choice to develop the form of the new instrument. Typically, two-dimensional views of the top of the violin were drawn. This view best helped to visualize the instrument. Mid-way into the form development process, a shift towards creating a more comfortable violin occurred. A strong emphasis was placed 85 upon a contoured form, especially where physical interaction occurs with the instrument. The rounded shape of a traditional violin creates an acute pressure point on the neck of musicians, felt mostly upon readjustment of the instrument. The form was developed to minimize this pressure point with a contoured surface matching the curve of the human neck. Figure 8.7 First Sketch of Developed Form Once the two-dimensional form had progressed further, full-size three- dimensional foam models were constructed to test for comfort, proportions and scale. The models were handled by two violinists to determine if the shape would be acceptable as an electric instrument. The new shape, particularly when in contact with the violinists? neck, was deemed to be comfortable, as well as an intriguing form for an instrument. 86 The rough foam models were successful in determining a solid direction for the form of the instrument and led to small tweaks to further refine the shape. Figure 8.8 Pink Foam Model of Violin Once the foam models were created, the neck and scroll area was given more attention. More sketches were done to develop a scroll that would match the aesthetic of the body. Many shapes were considered, but one idea surfaced that needed to be incorporated into the scroll and pegbox. To ease the process of stringing the instrument, and to provide a way of hanging the instrument, the pegbox needed to have four sides, rather than five. If the bottom of the peg box was removed it would create a hole that could be used for hanging the instrument, but it would also let the strings enter the pegs unhindered and the pegs could be accessed from two directions. 87 Figure 8.9 First Sketch of Developed Scroll The final scroll design retained the traditional stop used by violinists as a reference point, but removed excess weight from the end of the instrument. As the scroll was being developed, careful consideration was given as to the best way to display the music illumination system. The lights needed to glow rather than to shine, and it was also extremely important that the lights not be so bright that they would be a distraction to the violinist. Another factor that was highly considered was creating an instrument that would have a sophisticated aesthetic. The best way to incorporate all of these needs into the instrument was to have a wooden top and bottom, but to create the sides out of frosted acrylic. The frosted acrylic would work as a light diffuser, which would give the instrument its glow, and since the light would be predominately exiting the side of the instrument, the violinist would not be bothered. The wood top and bottom 88 would invoke a strong link to a traditional violin, but the shape and acrylic sides would separate it as a distinct instrument. Figure 8.10 Early Sketch of the Side Profile of the Instrument Once the aesthetics of the violin were mostly settled, attention was directed to how the instrument could be created. One particular problem that needed to be addressed was access to the electronics that would be installed inside the violin. To create the full glow of the instrument, LEDs and other electronics would need to be present in most of the instrument. Because of the widespread nature of the electronics it was decided that the best way to access the electrical components would be to have a removable back. A removable back led to more design features. The strings create a high amount of stress on the instrument because of the tension required to keep them in tune. One specific area of high stress would be the joint where the neck, top and sides of the body meet. Extra length was given to the area where the neck would be glued to the top of the body as can be seen in Figure 8.11, an early sketch of this idea. 89 Figure 8.11 Early Sketch of the Neck and Body Interface Also, to increase strength in the wooden top, a structural element similar to a bass bar was incorporated into the design. One final touch to the design was the addition of the ?f? holes. In keeping with the traditional-meets-breakthrough aesthetic, the shape of the new ?f? holes was taken from a portion of traditional ?f? holes. Violin makers typically place a small notch near the center of the ?f? holes. This notch is a cue as to the placement of the bridge. The placement notch inspired the ?f? holes for the new violin. 8.5 Three-Dimensional Computer Modeling A three-dimensional computer model was created to fully develop the form into a more realistic object. Three-dimensional modeling is used to create conceptual renderings, and is also a good way to experiment with how the parts of an object will be assembled to create the whole. Modeling allows these experiments to take place with no material waste, or time and money spent in the production process. Problems can sometimes be averted if a well thought out three-dimensional computer model is made. When modeling the violin, measurements were taken from the full size drawings and foam models and the basic shape was drawn in the computer. 90 Figure 8.12 Screenshot of Acrylic Piece Modeled Using Rhinoceros Care was taken to leave enough space for the electrical components, and the way each part of the violin would connect or join with its surrounding parts was carefully considered. Computer renderings were then created, using the model built in the three- dimensional computer modeling program, Rhinoceros. Figure 8.13 Computer Rendering Made Using Rhinoceros with Flamingo Plug-in 91 Figure 8.14 Computer Rendering Made Using Rhinoceros with Flamingo Plug-in Once a satisfactory appearance had been reached, the model was tweaked to make it usable to a Computer Numeric Controlled (CNC) router. 8.6 Creation of the Model 8.6.1 Non-Electronic A CNC router is a machine that takes a three-dimensional computer model and uses a multi-axis positioning system to carve the given material into the desired shape using a bit similar to bits used in a drill. The process takes many hours to complete because the bit removes very small amounts of material with each pass. Each piece was first tested using a soft foam material that allowed the CNC router to work faster. Once a suitable test model had been completed, parts were then made from the final materials: basswood, acrylic and maple. 92 Figure 8.15 CNC Router Figure 8.16 Foam ?Acrylic? Piece Test Fit with Wooden Top 93 Figure 8.17 CNC Router Making Acrylic Piece Figure 8.18 CNC Router Making Neck 94 When pieces are removed from the CNC router, there is still a significant amount of work that must be completed for the pieces to be smooth to the touch. The acrylic piece needed a large amount of sanding to give it the proper tactile feeling. The wood pieces required scraping and sanding, the neck being the piece that needed the most work done by hand. The neck was left intentionally unrounded in places so that the final shape could be finished by hand, giving it a less machined feel and appearance. Figure 8.19 Wooden Top in Bounding Box from CNC Router Figure 8.20 Neck in Bounding Box from CNC Router 95 Figure 8.21 Body Assembly Test Fit ? Before Sanding Once the pieces were smooth enough, stain and lacquer were applied to the wood pieces. The stain color used is similar to the color of traditional violins, but it was not achieved without many tests of different colors, combinations and types of stains. Figure 8.22 Staining 96 Next, the pieces were assembled. The neck was glued to the wooden top, and then the acrylic was glued to the neck and body. Finally, the fingerboard was glued to the neck. The bottom of the instrument was attached using screws going into aluminum standoffs embedded into the acrylic. Nine screws were used to ensure a proper fit to deter against warping. Figure 8.23 Gluing the Violin Together Once the violin was completely assembled the strings and bridge were installed. This step also involved the installation of the pegs and tailpiece. 97 8.6.2 Electronic The electronics of the music illumination system were a major challenge in the creation of the violin. A workable solution was finally reached after learning more about the Arduino Development Platform. The plan was to install five full-color, RGB LEDs on the interior of the violin, three in the center of the instrument and one in each end. These LEDs would be wired to a jack on the exterior of the instrument where a cable could be attached. The cable would run to a breadboard attached to the Arduino microcontroller that could be connected to a computer to be programmed. The cable also would bring power to the lights. Figure 8.24 Three Blue LED Portion of Electronics Schematic from Lab Chalkboard 98 Tests were conducted in the Electrical Engineering labs at Auburn University. When all of the kinks were resolved, the electrical components were assembled. Figure 8.25 Electrical Lab Work Holes had to be drilled into the acrylic piece to accept the jacks. One hole for the audio jack and one hole for the electronics jack were carefully drilled. The audio jack is a standard 1/4? electric instrument jack, but the jack for the electronics was uncharted territory. An Ethernet connection was selected because it had the correct number of individual wires, a durable connection, and cable could be found because of one electric guitar that uses the same connection for a different application. 99 Figure 8.26 Test Fit of Jacks Figure 8.27 First Test of Actual Electronic Components to be Installed Once each individual part of the electronics had been assembled, the LEDs could be installed into the instrument. Each component rests on a pad of acrylic, but is attached with a hook and loop fastening system so that removal for repair will be quick and simple. 100 Figure 8.28 Electronic Components Installed in the Acrylic 8.7 Semi-Working Model Although the goal of a fully functioning, real-time model was unreachable, an extremely convincing video was able to be made in which the violin appears to work in real-time. The video was made by creating audio recordings of the electric violin, analyzing the sound files using Tartini, then programming the Arduino to control the array of LEDs to respond to the sound file. Once this was completed, the video was created by ?lip synching? the original performance in time with the LEDs 101 Figure 8.29 Screenshot of Video 8.8 Introduction of the Vivisi Electric Violin 8.8.1 Overview of Results and Benefits The Vivisi electric violin is a musical instrument that produces a visualization of sound within the body of the instrument. Based on research of the medical phenomenon of synesthesia, the union of the senses, the Vivisi electric violin is able to display the rhythm, pattern and volume of music. The Vivisi electric violin has two main uses: one as a performance instrument and the other as a training tool. 102 As a performance instrument, the Vivisi electric violin allows a musician to create in both the auditory and visual realms. This adds interest for the audience, but also allows improvisational artists a new way of interacting with a musical instrument. As a training tool, the Vivisi electric violin helps to train young musicians to play in tune. From the body of the instrument, softly glowing lights give positive peripheral feedback to trainees when they play in tune. The Vivisi electric violin is the end result of the opportunities that came about during the research stage of the design process. The knowledge gained during the color survey led to a better understanding of the associations people have between color and musical tones. Directed by the color survey, each note on the chromatic scale that can be played on the violin was paired with a specific color. Notes that are octaves apart are represented by the same base color, but as pitch increases colors become whiter. The one-to-one color-note relationship allows the music played on the violin to be seen in the patterns and rhythms of the colors that emanate from the body of the instrument. The form of the violin is directly related to the comfort of the musician. A strong emphasis was placed upon a contoured form, especially where physical interaction occurs with the instrument. Traditional violins create an acute pressure point on the neck of musicians, felt mostly upon readjustment of the instrument. The form was developed to minimize this pressure point with a contoured surface matching the curve of the human neck. The asymmetric form of the instrument gives visual interest, but also creates a longer viewing surface for the color to be displayed upon, as well as adding extra space 103 for the on-board electronics. The ?f? holes are reminiscent of the traditional ?f? holes of a classical violin, but with a simpler, yet stronger visual dynamic. The Vivisi electric violin is a performance instrument, yet one that touches both the auditory and visual senses at once. Improvisational artists will find a new output for their creativity, and audiences will be highly engaged due to the direct integration of both the auditory and visual senses. Audience members who are hearing impaired will also find enjoyment in the performance. When used as a performance instrument, Vivisi has a set color output for any frequency that can be played. The same technology used to make the Vivisi electric violin an engaging performance instrument can be slightly changed to create a top-notch training tool for those who wish to learn a non-fretted instrument. Practicing non-fretted instruments, such as the violin, can be discouraging for beginners because of the long learning curve that occurs when learning to play in tune. The finger must be in just the right place for the correct note to be played. The Vivisi electric violin eases this process by providing positive feedback when notes are played in tune. Students will know they have played a note in tune when the violin glows and will be able to recognize what pitch they have played by the color they see. This ambient feedback is an improvement from watching an electronic tuner because it moves the action of in-song tuning from the center of attention to the periphery, allowing students to focus on reading sheet music or the more physical aspects of playing the instrument. The Vivisi electric violin is a new performance instrument and training tool that gives musicians a new way to interact with a violin, audiences a new way to observe a performance and students a new way to improve their skills. 104 8.8.2 Other Applications It has been explained that the technology used to make the Vivisi a compelling performance instrument can be changed slightly to create a helpful training tool for students of the violin. The potential for this tuning illumination system is great. The tuning illumination system can be removed from the body of the violin and encased in an attachment to a music stand. This product could have great appeal to students of any stringed instrument, but also other instruments where a physical action is required to tune a note, such as a trombone. The attachment would be an ambient display of real-time tuning, possibly glowing on the left side for flat notes, on the right side for sharp notes, and glowing brighter in the center for notes that are played in tune. The peripheral nature of the device would allow students to focus on sheet music, while giving gentle hints to help tune a current note. A tuning illumination system attached to a stand could be sold for a much lower price than an instrument, making the technology available to a wider audience. Other applications of the music illumination system include the creation of more Vivisi Instruments, such as a viola and a cello, or creating Vivisi Cubes to illuminate specific instruments in an orchestra or even for use attached to a solo instrumentalist?s looping system. The approach and process, detailed in this thesis paper, that was used in the design of the Vivisi electric violin, resulted in a quality product, backed by user-centered research, ergonomic studies, theoretical and practical knowledge, and determination. A similar approach and process can be applied to a variety of other products and is 105 particularly applicable to the design of ambient peripheral displays of information using colored light and to the design of musical instruments. 8.8.3 Images The following full-page images are presented to visually demonstrate the instrument and its attributes. 106 Figure 8.30 The Vivisi Electric Violin 107 Figure 8.31 The Vivisi Electric Violin in Concert 108 Figure 8.32 One-to-One Color-Note Relationship 109 Figure 8.33 What is the Color of Music? 110 Bibliography Ambient Devices. 2007. Ambient Devices, Inc. 27 January 2008 . Arduino. 2007. Arduino. 11 March 2008 . Barbera, Rich. ?Barbera Transducer Systems.? Barbera Transducers.com. 2007. 10 Mar. 2008. < http://www.barberatransducers.com/>. Beauchamp, G. D. ?Stringed Musical Instrument.? U.S. Patent 2,310,199. 9 February 1943. ---. ?Stringed Musical Instrument.? U.S. Patent 2,130,174. 13 September 1938. Bishop, Bainbridge. A Souvenir of the Color Organ, with Some Suggestions in Regard to the Soul of the Rainbow and the Harmony of Light. New York: The De Vinne Press, 1893. Berg, Richard E. and David G. Stork. The Physics of Sound. Englewood Cliffs, New Jersey: Prentice-Hall, 1982. Bowers, Faubion. The New Scriabin: Enigma and Answers. New York: St. Martin?s Press, 1973. Brougher, Kerry, Jeremy Strick, Ari Wiseman, and Judith Zilcher. Visual Music. London: Thames and Hudson, 2005. 111 Callopy, Fred. ?Rhythmic Light.? Rhythmic Light.com. 19 Nov. 2004. 16 Jan. 2008. . Colwell, Richard J. and Thomas W. Goolsby. The Teaching of Instrumental Music. 3rd ed. Upper Saddle River, New Jersey: Prentice Hall, 2002. Cytowic, Richard, E. The Man Who Tasted Shapes. New York: Putnam?s Sons, 1993. Dann, Kevin, T. Bright Colors Falsely Seen: Synaesthesia and the Search for Transcendental Knowledge. New Haven, Connecticut: Yale University Press, 1998. Farga, Franz. Violins and Violinists. 2nd ed. London: The Camelot Press, 1969. Fender, C. L. ?Electric Violin and Electro Magnetic Pickup Therefor.? U.S. Patent 3,003,382. 10 October 1961. Gerstner, Karl. The Forms of Color. Cambridge, Massachusetts: The MIT Press, 1986. Gill, Dominic (ed.) The Book of the Violin. New York: Rizzoli International Publications, 1984 Greenfield, Adam. Everyware: The Dawning Age of Ubiquitous Computing. Berkeley, California: New Riders, 2006. Goetschius, Percy. The Structure of Music. Westport, Connecticut: Greenwood Press, 1934. Hankins, Thomas L. and Robert J. Silverman. Instruments and the Imagination. Princeton, New Jersey: Princeton University Press, 1995. Harrison, John. Synaesthesia: The Strangest Thing. New York: Oxford University Press, 2001. 112 Head, Phineas de Thornley. ?Synaesthesia: Pitch-Colour Isomorphism In Rgb-Space?? Cortex 42 (2006): 164-174. Hull, Arthur Eaglefield. A Great Russian Tone-Poet: Scriabin. New York: AMS Press, 1918. Maconie, Robin. The Concept of Music. Oxford: Clarendon Press, 1990. ---. The Science of Music. Oxford: Clarendon Press, 1997. Mankoff, Jennifer and Anind Dey. "From Conception to Design, A Practical Guide to Designing Ambient Displays." Public and Situated Displays. Ed. Ohara, K and Churchill, E. Norwell, Massachusetts: Kluwer Academic Publishers, 2003. 210- 230. Mathews, Max V. ?Bimorphic Piezoelectric Pickup Device For Stringed Musical Instruments.? U.S. Patent 4,860,625. 29 August 1989. Matthews, Tara L. ?Designing and Evaluating Glanceable Peripheral Displays.? Diss. University of California at Berkeley, 2007. McLeod, Philip. ?Tartini.? Tartini: The Real-Time Music Analysis Tool. 2007. 8 Mar. 2008. . Olson, Harry, F. Music, Physics and Engineering. New York: Dover Publications, 1967. Peacock, Kenneth. "Instruments to Perform Color-Music: Two Centuries of Technological Experimentation." Leonardo Vol. 21, No. 4, (1988): 397-406. Pierce, John R. The Science of Musical Sound. New York: Scientific American Books, 1983. Raichel, Daniel R. The Science and Applications of Acoustics. New York: Springer- Verlag, 2000. 113 Robertson, Lynn C. and Noam Sagiv, (eds.) Synesthesia: Perspectives from Cognitive Neuroscience. New York: Oxford University Press, 2005. Saffer, Dan. Designing For Interaction: Creating Smart Applications and Clever Devices. Berkeley, California: New Riders, 2007. Schussler, John. ?Bowed Electricity.? Bowed Electricity.com. 2001. 8 Mar. 2008. < http://www.bowedelectricity.com/>. Shanet, Howard. Learn to Read Music. New York: Simon and Schuster, 1956. Smith, Jeff. ?David Leadbetter Swing Setter Review." The Sand Trap.com. 4 August 2005. 28 January 2008 . Stowell, Robin. The Cambridge Companion to the Violin. Cambridge: Cambridge University Press, 1992. Taylor, Charles. Exploring Music: The Science and Technology of Tones and Tunes. London: Institute of Physics Publishing, 1992. Tower, Beeke Sell. Klee and Kandinsky in Munich and at the Bauhaus. Ann Arbor, Michigan: UMI Research Press, 1978. Winckel, Fritz. Music, Sound and Sensation: A Modern Exposition. Trans. Thomas Binkley. New York: Dover Publications, 1967. Ward, Jamie, Brett Huckstep, and Elias Tsakanikos. ?Sound-Colour Synaesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common To Us All?? Cortex 42 (2006): 264-280. Weiser, M and J.S. Brown. "Designing Calm Technology." PowerGrid Journal (1996): 23 January 2008 114 ---. "The Coming Age of Calm Technology." Beyond Calculation: The Next Fifty Years in Computing. Ed. Denning, Peter J. and Metcalfe, Robert M. New York: Copernicus, 1997. 75-85. Wood, Alexander. The Physics of Music. Revised. J. M. Bowsher. London: Chapman and Hall, 1975.