| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| |
| <html> |
| |
| <head> |
| <meta http-equiv="content-type" content="text/html;charset=iso-8859-1"> |
| <title>StarOffice Calc v. MS Excel, 2001</title> |
| </head> |
| |
| <body bgcolor="#ffffff"> |
| <div align="right"> |
| |
| </div> |
| <table width="100%" border="0" cellspacing="0" cellpadding="0"> |
| <tr> |
| <td width="22"></td> |
| <td colspan="2"> |
| <div align="right"> |
| by <a href="http://www.ischool.berkeley.edu/people/alumni/susannebrokaw">Susanne Eklund</a>, Michal Feldman, and Mary Trombley</div> |
| </td> |
| </tr> |
| <tr> |
| <td width="22"></td> |
| <td colspan="2"> |
| <h1>StarOffice Calc v. MS Excel<a href="../../../index.html"><img src="../../../_media/uxooo-logo.gif" alt="ux.ooo home page" width="122" height="80" align="right" border="0"><br> |
| </a>Improving the Usability of an Open Source Spreadsheet Application</h1> |
| <p>12 December 2001<br> |
| InfoSys 271 - Quantitative Research Methods for Information Management<br> |
| Professor Rashmi Sinha<br> |
| School of Information Management and Systems<br> |
| University of California, Berkeley<br> |
| </p> |
| <p><a href="Eklund2001.pdf">Download PDF Version</a> <sup>3450 KB</sup></p> |
| <p></p> |
| </td> |
| </tr> |
| <tr> |
| <td width="22"></td> |
| <td width="88%"> |
| <h2></h2> |
| <h2 align="left">Table of Contents</h2> |
| <hr size="1" width="100%"> |
| <div align="left"> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742108">Executive Summary<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">3</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742109">Introduction<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">4</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742112">Method<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">6</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742115">Experimental Design<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">9</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742117">Results<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">13</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742119">Usability Problems and Design Recommendations<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">29</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742122">References<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">33</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="#_Toc532742123">Appendices<span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">34</span></a></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="codebook_for_starcalc_data.doc">Data Codebook</a><span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">34</span></span></p> |
| <p class=MsoToc1><span style="color:blue"><a href="calc_study_dataset.sav">SPSS Dataset</a><span> </span><span style="color:windowtext;display:none;text-decoration:none; |
| text-underline:none">34</span></span></p> |
| </div> |
| <span style='font-size:14.0pt;font-family:"Arial Black"; |
| |
| letter-spacing:-.5pt;'><br clear=all style="page-break-before:always"> |
| </span> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc532742108">Executive Summary</a></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <p class=MsoBodyText align=left style="text-align:left">In this report, we evaluate the beta release of StarOffice Calc 6.0, the spreadsheet application contained in Sun Microsystems' StarOffice open source software suite. We compare StarOffice Calc to Microsoft Excel 2000.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Our study has the following objectives:</p> |
| <ul> |
| <li class=MsoBodyText style="text-align:left; |
| ">To determine which parts of StarOffice Calc's interface support or hinder the activities of current Excel users. |
| <li class=MsoBodyText style="text-align:left; |
| ">To compare user satisfaction ratings for the two systems. |
| <li class=MsoBodyText style="text-align:left; |
| ">To develop design suggestions for the StarOffice Calc development team based on our quantitative and qualitative data. |
| </ul> |
| <p class=MsoBodyText align=left style="text-align:left">We tested twelve graduate students from the University of California, Berkeley.? Our participants were all current Excel users who worked with Excel at least once a month for various business and educational purposes.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Our study evaluated common spreadsheet functions in four main categories: Basic Formatting, Formulas, Advanced Formatting, and Charts. We presented our participants with two scenarios, one for Excel and one for StarOffice Calc.? Each scenario consisted of four discrete stages corresponding to the four categories. ?Each stage required the participants to complete a series of tasks.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Our participants rated almost all of the 15 tasks slightly easier to complete in Excel than in StarOffice Calc, but the reported difference between the two systems was only significant in two tasks: merging cells and inserting charts, both of which were rated significantly easier in Excel.? Other tasks that were particularly difficult to perform in StarOffice Calc included inserting a formula, drawing and formatting shapes, and choosing data ranges for charts. ?Only one task was easier in StarOffice Calc: changing the alignment of text in cells.? However, the difference between Calc and Excel was not statistically significant for this task.</p> |
| <p class=MsoBodyText align=left style="text-align:left">When participants rated each system separately, Excel scored slightly higher than StarOffice Calc on all measures. Differences were significant for the following measures: usefulness, enjoyment, ability to figure out without help, and ease of understanding icons.? Average system satisfaction was significantly higher in Excel than in StarOffice Calc.? When users directly compared the systems, they reported, on average, a slight preference for Excel.</p> |
| <p class=MsoBodyText align=left style="text-align:left">While Excel outperformed StarOffice Calc in almost all of our measures, it should be noted that the differences were small and quite often statistically insignificant.? Moreover, 92 percent of participants expressed willingness to switch to StarOffice Calc under the current price structure.</p> |
| <p class=MsoBodyText>After identifying four major problem areas in the StarOffice Calc interface, we use a severity scale and our qualitative observations to discuss the problems and suggest design improvements.</p> |
| <span style='font-size:14.0pt;font-family:"Arial Black"; |
| |
| letter-spacing:-.5pt;'><br clear=all style="page-break-before:always"> |
| </span> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc457293991"></a><a name="_Toc457293989"></a><a name="_Toc465752685"></a><a name="_Toc465841289"></a><a name="_Toc532742109"><span><span>Introduction</span></span></a></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <h3 style="text-align:justify"><a name="_Toc465752686"></a><a name="_Toc465841290"></a><a name="_Toc532742110"><span>Full Product Description</span></a></h3> |
| <p class=MsoBodyText align=left style="text-align:left">This test evaluated the beta release of StarOffice Calc 6.0, a spreadsheet application originally produced by Sun Microsystems.? Calc is part of Sun?s StarOffice suite, an open source collection of office software applications that competes with Microsoft Office. We compared Calc to Microsoft?s Excel 2000 spreadsheet application.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Our study evaluated frequently-used spreadsheet functions in four main categories: basic formatting, formulas, advanced formatting, and charts.? The categories and associated tasks are outlined in <span style='"REF _Ref532710543 \\r \\h  \\* MERGEFORMAT"'>Table 1</span>.</p> |
| <p class=TableHeader><a name="_Ref532710543">Table 1.<span style='font:7.0pt "Times New Roman"'> </span>Task Structure</a></p> |
| <table border=1 cellspacing=0 cellpadding=0> |
| <tr> |
| <td width=151 valign=bottom bgcolor="#D9D9D9" class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Category</b></p> |
| </td> |
| <td width=415 valign=bottom bgcolor="#D9D9D9" class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Associated Tasks</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=151 valign=top class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Basic Formatting</b></p> |
| </td> |
| <td width=415 valign=bottom class="Normal"> |
| <ol start=1 type=1> |
| <li style=" ">Shifting cell contents to another cell |
| <li style=" ">Inserting empty rows and columns |
| <li style=" ">Merging cells |
| <li style=" ">Converting numbers into currency format |
| <li style=" ">Expanding the width or height of a row or column |
| <li style=" ">Changing the alignment of text in a cell |
| </ol> |
| </td> |
| </tr> |
| <tr> |
| <td width=151 valign=top class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Formulas</b></p> |
| </td> |
| <td width=415 valign=bottom class="Normal"> |
| <ol start=7 type=1> |
| <li style=" ">Inserting a formula |
| <li style=" ">Applying a formula to multiple cells |
| </ol> |
| </td> |
| </tr> |
| <tr> |
| <td width=151 valign=top class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Advanced Formatting </b></p> |
| </td> |
| <td width=415 valign=bottom class="Normal"> |
| <ol start=9 type=1> |
| <li style=" ">Changing font color |
| <li style=" ">Adding borders to cells |
| <li style=" ">Adding background color to cells |
| <li style=" ">Drawing and formatting shapes |
| </ol> |
| </td> |
| </tr> |
| <tr> |
| <td width=151 valign=top class="Normal"> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Charts</b></p> |
| </td> |
| <td width=415 valign=bottom class="Normal"> |
| <ol start=13 type=1> |
| <li style=" ">Inserting a chart |
| <li style=" ">Selecting the data range for a chart |
| <li style=" ">Adding titles to a chart |
| </ol> |
| </td> |
| </tr> |
| </table> |
| <p class=MsoBodyText align=left style="text-align:left">Sun promotional materials <a href="#reference1">[1]</a> indicate that the StarOffice suite has been designed for "small and home office users who need an alternative productivity suite and...education and enterprise users who work in mixed-platform environments".?</p> |
| <p class=MsoBodyText align=left style="text-align:left">From this broad user group, we decided to test current Excel users. Our participants regularly use Excel for various business and educational purposes including accounting, budgeting, financial and statistical analysis, and chart creation.<span>? </span></p> |
| <h3 style="text-align:justify;"><a name="_Toc457293992"></a><a name="_Toc465752687"></a><a name="_Toc465841291"></a><a name="_Toc532742111"><span>Test Objectives</span></a></h3> |
| <p class=MsoBodyText align=left style="text-align:left">Our usability test had several objectives:</p> |
| <p class=MsoBodyText align=left style="text-align:left;"><b>Objective 1:? </b>Investigate which parts of StarOffice Calc's interface <span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'><span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'>support or hinder the activities of current Excel <span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'>users.</span></span></span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">Which parts of the interface are troublesome for users?</span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">Which specific tasks are troublesome for users?</span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">In which areas do participants spend significantly more time completing tasks in StarOffice Calc than in Excel?</span></p> |
| <p class=MsoBodyText align=left style=" |
| text-align:left;"><b>Objective 2:? </b>Compare <span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'><span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'>user <span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'>satisfaction ratings for the two system<span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'><span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1315'>s.</span></span></span></span></span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">Are participants significantly more satisfied with one system than the other? </span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">Do participants enjoy using one system more than the other?</span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">Were users satisfied with file conversions between the two formats?</span></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| "><span style="font-family:Symbol;letter-spacing:0pt;">?<span style='font:7.0pt "Times New Roman"'> </span></span><span style="letter-spacing:0pt">After using StarOffice Calc to perform basic tasks, are the participants willing to switch from Excel to StarOffice Calc? </span></p> |
| <p class=MsoBodyText align=left style=" |
| text-align:left;"><b>Objective 3:? </b>Develop <span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1316'>design suggestions for the StarOffice Calc<span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1316'> development team based on our qua<a name="_Toc457293993"></a><a name="_Toc465752688"></a><a name="_Toc465841292"><span><span class=msoChangeProp style='"Susanne M\. Eklund" 20011211T1316'>ntitative and qualitative data.</span></span></a></span></span></p> |
| <span style='font-size:14.0pt;font-family:"Arial Black"; |
| |
| letter-spacing:-.5pt;'><br clear=all style="page-break-before:always"> |
| </span> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><span><a name="_Toc532742112">Method</a></span></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <h3 style="text-align:justify"><a name="_Toc457293994"></a><a name="_Toc465752689"></a><a name="_Toc465841293"></a><a name="_Toc532742113"><span>Participants</span></a></h3> |
| <p class=MsoBodyText align=left style=" |
| text-align:left">Our twelve participants were graduate students from the Haas Business School and the School of Information Management and Systems (SIMS) at the University of California, Berkeley.? We selected these participants through solicitations to general mailing lists at Haas and SIMS.? Their ages ranged from 22 to 41, with an average age of about 27.? Our participants were all current users of Excel who worked with Excel at least once a month.? The majority of our participants (92%) used Excel at least weekly.</p> |
| <p class=MsoBodyText align=left style=" |
| text-align:left">The average participant:</p> |
| <p class=MsoBodyText><span style="font-family:Symbol">?<span style='font:7.0pt "Times New Roman"'> </span></span>Used Excel slightly more than once a week</p> |
| <p class=MsoBodyText><span style="font-family:Symbol">?<span style='font:7.0pt "Times New Roman"'> </span></span>Created charts and graphs in Excel once a month</p> |
| <p class=MsoBodyText><span style="font-family:Symbol">?<span style='font:7.0pt "Times New Roman"'> </span></span>Used calculation functions more than once a month</p> |
| <p class=MsoBodyText><span style="font-family:Symbol">?<span style='font:7.0pt "Times New Roman"'> </span></span>Formatted tables more than once a week</p> |
| <p class=MsoBodyText><span style="font-family:Symbol">?<span style='font:7.0pt "Times New Roman"'> </span></span>Used advanced features, like macros, more than once a year.</p> |
| <p class=MsoBodyText align=left style=" |
| text-align:left">Our participant sample differs from a random sample of the population in several ways.? Since we recruited participants from a university, our sample has a high average level of education.? Our participants most likely have high levels of computer knowledge and easier access to technology.? Finally, they currently use spreadsheets in an educational environment rather than solely in the home or in the workplace.? Our participant?s average age most likely also differs from the population.</p> |
| <p class=MsoBodyText align=left style="text-align:left"><span style="letter-spacing:0pt">Table 2</span><span style="letter-spacing: |
| 0pt"> presents a summary of </span>our<span style="letter-spacing:0pt"> participants? salient characteristics. </span></p> |
| <p class=TableHeader><a name="_Ref532642335">Table 2.<span style='font:7.0pt "Times New Roman"'> </span>Participant Matrix</a></p> |
| <table border=1 cellspacing=0 cellpadding=0 width=575> |
| <tr> |
| <td width=123 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText><a name="_Toc457293995"></a><a name="_Toc465752690"></a><a name="_Toc465841294"><span><b>Participant No.</b></span></a></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>Age</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>Gender</b></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=right style=" |
| text-align:right"><span><b>Frequency of Excel Usage </b></span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>1</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=bottom class="Normal"> |
| <p align=center style="text-align:center"><span><span>34</span></span><span><span style=" |
| "></span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p align=center style="text-align:center"><span><span>M</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>2</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=bottom class="Normal"> |
| <p align=center style="text-align:center"><span><span>29</span></span><span><span style=" |
| "></span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p align=center style="text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Daily</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>3</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>22</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>M</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>4</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>23</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>5</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>29</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>M</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Daily</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>6</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>26</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>M</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>7</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>41</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>8</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>24</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Monthly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>9</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>29</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Monthly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>10</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>23</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>M</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Weekly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>11</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>23</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Monthly</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=123 class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><b>12</b></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>26</span></span></p> |
| </td> |
| <span></span> |
| <td width=124 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span><span>F</span></span></p> |
| </td> |
| <span></span> |
| <td width=204 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style=" |
| text-align:center"><span>Daily</span></p> |
| </td> |
| <span></span></tr> |
| </table> |
| <b><span style='font-size:13.0pt;font-family:Arial;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b><br> |
| <h3><span><a name="_Toc532742114">Context of Product Use in the Test</a></span></h3> |
| <h4><a name="_Toc457293996"></a><a name="_Toc465752691"></a><a name="_Toc465841295"><span>Tasks</span></a></h4> |
| <p class=MsoBodyText align=left style="text-align:left">We presented our participants with two major scenarios, one for Excel and one for StarOffice Calc.? These scenarios included four discrete stages: basic formatting, adding formulas to cells, advanced formatting, and adding charts.?</p> |
| <p class=MsoBodyText align=left style="text-align:left">Since we felt that appropriate task selection was important to the success of our study, we made use of several resources in choosing a task set.? We examined each spreadsheet?s Help sections and marketing materials <a href="#reference1">[1]</a> to determine what the developers felt were the most important tasks.? We also asked current Excel users to describe the tasks they performed frequently.? Finally, we used our personal experience with Excel to help shape the tasks.</p> |
| <p class=MsoBodyText align=left style="text-align:left">At the test commenced, the computer screen displayed a set of base data in either Excel or StarOffice Calc.? The test administrator then showed the participant a stage illustration, a sheet of paper that showed the base data with significant formatting or calculation alterations (see Appendix D for stage illustrations).? The administrator asked the participant to change the base data on the screen to make it look like the data on the paper.? Once the participant had completed all the tasks in the stage, the administrator showed him or her the same data set with more alterations.? The participant again manipulated the base data until it appeared like the data on the sheet.? After completion of all four stages, the participant was asked to repeat a similar set of tasks in the other spreadsheet with slightly different data.</p> |
| <p class=MsoBodyText>We considered a task complete when the participant?s spreadsheet substantially resembled the stage illustration with respect to the task.? If a participant did not notice that s/he had neglected to complete a certain task, we prompted him/her to perform the task.? We considered a stage complete when the participant had either completed all tasks or indicated that s/he was not going to finish the remaining task(s).? The participants themselves decided when to abandon a particular task.?</p> |
| <h4>Test Facility and Computing Environment</h4> |
| <p class=MsoBodyText align=left style="text-align:left">The participants were tested in a computer lab at the University of California, Berkeley.? The lab had a low to medium noise level during the test.? This environment is equipped with cubicles and standard office equipment; it resembles a workplace environment.? The test equipment and setup is described in <span style='"REF _Ref532711400 \\r \\h  \\* MERGEFORMAT"'>Table 3</span>.</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <p class=TableHeader><a name="_Ref532711400">Table 3.<span style='font:7.0pt "Times New Roman"'> </span>Test Equipment and Setup</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=169 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><b>Item</b></p> |
| </td> |
| <td width=390 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><b>Specifications</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Computer model</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>Participants were tested on HP Kayak model XU800 computers, equipped with Windows OS 2000.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>StarOffice Calc Settings</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>Our participants interacted with the StarOffice Calc settings identical to the default settings that are distributed by Sun Microsystems.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Excel Settings</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>Excel settings were identical for each participant, and they included the drawing toolbar on the bottom of the main window.???</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Screen Size</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>22 inch monitor</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Screen Resolution</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>1280 by 1024 pixel resolution</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Color Setting</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>50% red, 50% green, and 92% blue</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p class=MsoBodyText>Input Devices</p> |
| </td> |
| <td width=390 valign=top class="Normal"> |
| <p class=MsoBodyText>HP standard keyboard</p> |
| </td> |
| </tr> |
| </table> |
| <h4><a name="_Toc457293999"></a><a name="_Toc465752694"></a><a name="_Toc465841298"><span>Test Administrator Tools</span></a></h4> |
| <p class=MsoBodyText align=left style="text-align:left">All questions to test participants were administered orally.? Laptops with Excel spreadsheets were used to record the participants? actions, responses to questions, time spent on each stage, verbal comments, and qualitative behavioral observations.? The question responses were recorded in a standard Excel spreadsheet; the other data was captured using an Excel logger macro.??</p> |
| <br clear=all style="page-break-before:always"> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc457294014"></a><a name="_Toc465752708"></a><a name="_Toc465841312"></a><a name="_Toc532742115"><span>Experimental Design</span></a></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <p class=MsoBodyText align=left style="text-align:left"><span>We tested our survey design on two pilot participants.? We revised our study design, task set, and questions based on the results of these pilot tests.? The following section describes our final design.</span></p> |
| <h4><span>Logical Design</span></h4> |
| <p class=MsoBodyText align=left style="text-align:left"><span>We divided our participants into two groups: half of them tested Excel first and StarOffice Calc second, and the other half tested StarOffice Calc first and Excel second. ?Using this design, we attempted to control two major effects: the fatigue effect and the learning effect. In order to control for gender, we divided the men and women between the groups as evenly as we could, given that we had more female participants than male.? The group design is illustrated in <span style='"REF _Ref532712476 \\r \\h  \\* MERGEFORMAT"'>Table 4</span>.? </span></p> |
| <p class=TableHeader><span><a name="_Ref532712476">Table 4.<span style='font:7.0pt "Times New Roman"'> </span>Group Design</a></span></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=135 valign=top bgcolor="#E0E0E0" class="Normal"><span> </span></td> |
| <span></span> |
| <td width=125 bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><span><b>Gender</b></span></p> |
| </td> |
| <span></span> |
| <td width=153 bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><span><b>First Spreadsheet</b></span></p> |
| </td> |
| <span></span> |
| <td width=153 bgcolor="#E0E0E0" class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><span><b>Second Spreadsheet</b></span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=135 valign=top class="Normal"> |
| <p class=MsoBodyText align=right style="text-align:right"><span><b>Group 1 </b></span></p> |
| </td> |
| <span></span> |
| <td width=125 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><span>2 Men, 4 Women</span></p> |
| </td> |
| <span></span> |
| <td width=153 valign=top class="Normal"> |
| <p class=MsoBodyText><span>Excel</span></p> |
| </td> |
| <span></span> |
| <td width=153 valign=top class="Normal"> |
| <p class=MsoBodyText><span>StarOffice Calc</span></p> |
| </td> |
| <span></span></tr> |
| <tr> |
| <td width=135 valign=top class="Normal"> |
| <p class=MsoBodyText align=right style="text-align:right"><span><b>Group 2</b></span></p> |
| </td> |
| <span></span> |
| <td width=125 valign=top class="Normal"> |
| <p class=MsoBodyText align=center style="text-align:center"><span>3 Men, 3 Women</span></p> |
| </td> |
| <span></span> |
| <td width=153 valign=top class="Normal"> |
| <p class=MsoBodyText><span>StarOffice Calc</span></p> |
| </td> |
| <span></span> |
| <td width=153 valign=top class="Normal"> |
| <p class=MsoBodyText><span>Excel</span></p> |
| </td> |
| <span></span></tr> |
| </table> |
| <p class=MsoBodyText align=left style="text-align:left"><span>We used slightly different data sets for each system.? We wanted the data sets to be similar enough that participants would perform the same tasks on each data set, but different enough that participants would be sufficiently challenged.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>While participants performed the same kinds of tasks within each data set, we did not restrict the ordering of the tasks within each stage.? For example, one stage required users to add empty columns, merge cells, and move data from cell to cell.? Participants could perform these tasks in any order they desired.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Asking users to repeat similar tasks with the second system increased the risk of a learning effect, because subjects became familiar with the tasks when they completed them in the first system. ?Our experiment design, with opposite ordering for the two user groups, is intended to counter these learning effects.</span></p> |
| <h4><span>Variables</span></h4> |
| <p class=MsoBodyText align=left style="text-align:left"><span>The following tables describe the measures we set out to capture and the variables we used to operationalize the measures.? </span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Table 5 details data we gathered through direct questioning.?? The complete question set can be found in </span><span>Appendix A</span><span>.? The scales used for each measure are summarized in <span style='"REF _Ref532719007 \\r \\h  \\* MERGEFORMAT"'>Table 6</span> and also included in their entirety in </span><span>Appendix E</span><span>.? </span></p> |
| <img src="images/table5.gif"><br> |
| <p class=TOCBase style="margin-bottom:0in;margin-bottom:.0001pt;line-height: |
| normal; |
| tab-stops:.5in"><span>The following scales are seven-point Likert scales.? To view the complete question scales, refer to </span><span>Appendix E</span><span>.</span></p> |
| <p></p> |
| <img src="images/table6.gif"><br> |
| <p class=MsoBodyText><span>Table 7 describes the measures that we observed directly.</span></p> |
| <p></p> |
| <img src="images/table7.gif"><br> |
| <p class=MsoBodyText><span>We also logged errors committed by the participants with the intention of analyzing this data quantitatively.? However, errors significantly differed from each other in quality and severity, and we did not have a well-defined definition of errors at the beginning of the test.? Therefore, we decided not to include the error count in our analysis.? Instead, comments about errors appear in the qualitative sections of our results.<a name="_Toc457294002"></a><a name="_Toc465752696"></a></span></p> |
| <br> |
| <br> |
| <span><span><span><a name="_Toc532742116"> |
| <h3>Test Procedure</h3> |
| </a></span></span>?</span> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Two testers, the administrator and the recorder, participated in each test. The administrator read the script and instructed the subject throughout the test. The recorder asked the participant questions at the end of each stage and recorded the answers. In addition, the recorder ?timed each stage and logged observations, comments, and behavior.? All data were recorded in Excel spreadsheets. </span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Our test was conducted under the supervision of the Human Subjects Committee at the University of California, Berkeley.? Accordingly, participants read and signed a consent form at the beginning of the test.? The full consent form is included in </span><span>Appendix B</span><span>. ?</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Participants were then given general test instructions.? They were asked to concentrate on the tasks and not to think aloud; however, any comments they made during the test were recorded. ?We instructed the participants to do their best, but we allowed them to stop a task at about the point they would normally abandon it if they were using a spreadsheet program in real life.? </span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>All participants were given five minutes to explore StarOffice Calc before they started to perform the Calc tasks.? For Group 1 participants, who tested Excel first, this period fell between the two systems; for Group 2 participants, who tested Calc first, it was the very first step of the test.? We hoped that this exploration period would help acclimate users to the system and allow them to become familiar with system functions and features before timing began.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Before starting each system, the participants were given printed sheets showing screenshots of data arranged in a spreadsheet.? The participants were given a starter file of raw data and asked to make the data look like it did in the printed copy.? When all changes were completed, the recorder administered task rating questions to the participant and logged the answers. Participants were provided with a printed scale with which to answer the questions (scale included in </span><span>Appendix E</span><span>). ?The participant then moved on to the next stage.? Participants completed four stages in StarOffice Calc and four stages in Excel, answering general questions about the system after each system?s stages were complete.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>On occasion, the administrator interrupted a participant when he or she was clearly unable to complete the task in a reasonable amount of time.? In these cases, we marked the task as incomplete and moved on.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>After both systems were tested and evaluated, the recorder asked some final comparison and summary questions.? The participants were then given the chance to express their feelings about the systems and converse freely with the testers.</span></p> |
| <p class=MsoBodyText align=left style="text-align:left"><span>After completion of the test, the participants were compensated for their time with a bookstore gift certificate worth twelve dollars.</span></p> |
| <h4><span>Participant Instructions</span></h4> |
| <p class=MsoBodyText align=left style="text-align:left"><span>Both general and individual task instructions for the participants were read verbatim from a script, which can be found in </span><span>Appendix C</span><span>.? The list of questions can be found in </span><span>Appendix A</span><span>, and the illustrations for each task stage can be found in </span><span>Appendix D</span><span>.</span></p> |
| <br clear=all style="page-break-before:always"> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><span><a name="_Toc532742117">Results</a></span></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <h3><a name="_Toc465752709"></a><a name="_Toc465841313"></a><a name="_Toc532742118"><span>Data Analysis</span></a></h3> |
| <p class=MsoBodyText align=left style="text-align:left">For definitions of our measures and variables, please refer to the previous Variables section under Experimental Design.</p> |
| <h4>Order Effects</h4> |
| <p class=MsoBodyText align=left style="text-align:left">As described earlier, participants were divided by gender and then placed randomly into two test groups containing roughly the same number of men and women.? Participants with System Order 1 tested Excel first and then StarOffice Calc; participants with System Order 2 tested Calc first and then Excel.? We analyzed the differences between System Order groups to be sure they are not significantly different.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Our analysis of System Order consisted of four multivariate ANOVA tests examining individual task ratings, time measures, system ratings, and overall comparison.? The independent variable for each ANOVA was System Order, and the dependent variables are explained in <span style='"REF _Ref532622286 \\r \\h  \\* MERGEFORMAT"'>Table 8</span>.</p> |
| <p class=TableHeader><a name="_Ref532622286">Table 8.<span style='font:7.0pt "Times New Roman"'> </span>Variables included in each ANOVA</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=283 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>ANOVA</b></p> |
| </td> |
| <td width=284 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Variables included</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=283 valign=top class="Normal"> |
| <p class=TOCBase style="line-height: |
| normal;">Task ratings</p> |
| </td> |
| <td width=284 valign=top class="Normal"> |
| <p>Task Ratings for all Excel tasks and all</p> |
| <p>??? StarOffice Calc tasks (30 variables)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=283 valign=top class="Normal"> |
| <p>Time measures</p> |
| </td> |
| <td width=284 valign=top class="Normal"> |
| <p>Stage Time for 4 stages, both Excel and??</p> |
| <p>StarOffice Calc (8 variables)</p> |
| <p>Excel Time - All Stages</p> |
| <p>StarOffice Calc Time - All Stages</p> |
| <p>Total Time - All Stages</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=283 valign=top class="Normal"> |
| <p>System ratings</p> |
| </td> |
| <td width=284 valign=top class="Normal"> |
| <p>General system evaluation questions for each system (16 variables)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=283 valign=top class="Normal"> |
| <p>Overall comparison ratings</p> |
| </td> |
| <td width=284 valign=top class="Normal"> |
| <p>Comparison questions administered at end of test for both systems together (5 variables)</p> |
| </td> |
| </tr> |
| </table> |
| <p class=MsoBodyText align=left style="text-align:left">The grand ANOVA results indicate that there are no significant differences between the groups with respect to task ratings (<i>F</i> (8,1) = 4.29; <i>p</i> > .05, n.s.), time measures (<i>F</i> (8,2) = 1.37; p > .05, n.s.), system ratings (<i>F</i> (10,1) = 11.88; <i>p</i> > .05, n.s.), or overall comparison ratings (<i>F</i> (5,6) = 0.72; <i>p</i> > .05, n.s.).?</p> |
| <p class=MsoBodyText align=left style="text-align:left">We therefore conclude that the fatigue and system order did not affect the data significantly, and that the groups are similar enough to be analyzed together from this point forward.</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <h4>Task Analysis</h4> |
| <p class=MsoBodyText align=left style="text-align:left">A primary objective of this study was to identify areas in the StarOffice Calc interface that negatively affected the performance of our participants and to suggest ways to improve the interface.? This section examines the results of all tasks and identifies tasks that are problematic in Calc. We then analyze these problematic tasks quantitatively and qualitatively and suggest ways to improve the interface to better support the tasks.?</p> |
| <p class=MsoBodyText align=left style="text-align:left">In the following section, we examine the Task Rating variable extensively.? Each user was asked to rate each task immediately after finishing the stage in which the task was attempted.? Participants were asked to rate the difficulty level of the task on the following scale:</p> |
| <p class="MsoBodyText"><b>Very difficult????????????????????????????????? ????? ?? Neutral??????????????? ??????????? ??????????? ??????????? ???????? Very easy</b></p> |
| <hr size="1" width="75%" align="left"> |
| <br> |
| 1??????????????????????? 2??????????????????????? 3??????????????????????? 4??????????????????????? 5??????????????????????? 6?????? ?????????????????7</p> |
| <p class=MsoBodyText align=left style="text-align:left">Each participant had two Task Ratings for each task, one in Excel and one in StarOffice Calc.?</p> |
| <p class=MsoBodyText align=left style="text-align:left">We examine each stage in turn, looking briefly at general trends; we examine the differences between the Excel Task Ratings and StarOffice Calc Task Ratings for each task, validate the differences statistically, consider the results in conjunction with our qualitative data, and suggest design improvements.</p> |
| <p class=MsoBodyText align=left style="text-align:left">It should also be noted that at times, users were unable to complete a given task because they indicated that in a normal situation they would be too frustrated to continue.?? Non-completion occurred twice in Excel (Tasks 12 and 14) and six times in StarOffice Calc (Tasks 3, 12, 13, and 14).? The non-completions are discussed in more detail in their respective task sections.</p> |
| <h5>Stage 1 ? Basic Formatting Tasks</h5> |
| <p class=MsoBodyText align=left style="text-align:left">In Stage 1, participants were asked to add basic formatting to a financial spreadsheet.? The stage consisted of six individual tasks, detailed in <span style='"REF _Ref532547354 \\r \\h  \\* MERGEFORMAT"'>Figure 1</span> below.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Overall, participants found this stage easier to complete in Excel.? The average Excel Task Rating was 6.26 (<i>s</i> = 0.71), while the average Calc Task Rating was 5.73 (<i>s</i> = 0.69).? A t-test confirmed that these means differ significantly (<i>t</i> (11) = 2.81; <i>p</i> < .05).????</p> |
| <p class=MsoBodyText align=left style="text-align:left">A quick visual examination of the mean Task Ratings in <span style='"REF _Ref532547354 \\r \\h  \\* MERGEFORMAT"'>Figure 1</span> suggests that most tasks align with the overall averages, scoring slightly higher for Excel than for Calc.? Tasks 1,2,4, and 5 are rated an average of 5.5% more difficult in Calc than in Excel.</p> |
| <br> |
| <img src="images/figure1.gif"><br> |
| <p class=MsoBodyText align=left style="text-align:left">Two tasks display different patterns than the rest.? Task 6, changing alignment of text in cells, is the only task rated easier for StarOffice Calc, while Task 3, merging cells, appears to be rated much easier in Excel.? Because these tasks stand out from the rest, we examined each individually.</p> |
| <h6>Task 3: Merging cells</h6> |
| <p class=MsoBodyText align=left style="text-align:left">A paired samples t-test confirmed that the apparent difference between the Excel and StarOffice Calc Task Ratings is indeed significant (<i>t</i> (10) = 3.30; <i>p</i> < .01) for Merging Cells.? In addition, one participant was not able to complete this task at all, 6 participants had to use the Help Menu to find the Merge function, and 11 of our participants received an interruptive error message at one point in the process.???</p> |
| <p class=MsoHeading7><b>Explanation of the Problem</b></p> |
| <p class=MsoBodyText align=left style="text-align:left">To merge cells in Excel, users have the choice of clicking on a "Merge and Center" button on the toolbar or going to the Format Cells dialog box, choosing the alignment tab, and checking the "Merge Cells" checkbox.? All users had performed this task before, and the majority used the Format Cells dialog box to perform the task.? When using StarOffice Calc, they tended to look in the same place, carefully examining each tab in Format Cells to find the right checkbox.? When they did not find "Merge Cells," they did not know where else to look.? Most users eventually found the item in the Format menu, but some remarked that selecting "define" was somewhat unintuitive.?</p> |
| <p class=MsoBodyText align=left style="text-align:left">We did observe that once users discover where the function is located in StarOffice Calc, they did not mind its placement; the burden lay primarily in the initial learning curve.</p> |
| <p class=MsoBodyText align=left style="text-align:left">In addition, one step required users to re-do a previously completed merge.? Almost all users got an error message when they tried to redefine an existing merged cell.? While it was not a fatal error, the message tended to annoy users.?? One participant remarked, ?You have to undo merge to redo it?? That?s stupid!?</p> |
| <p class=MsoBodyText align=left style="text-align:left"><b>Recommended Design Change<br> |
| </b>To fix the Merge Cells problem, we suggest retaining the menu command but renaming it to better match user expectations.? ?Merge cells? should appear if the current selection is un-merged, and ?Un-merge cells? should appear if the current selection is merged.? This way the user will not have an occasion to receive an error message.? We also suggest repeating the Merge Cells option in the Cell Format dialog, since people tend to look there first.</p> |
| <h6>Task 6: Changing alignment of text in cells</h6> |
| <p class=MsoBodyText align=left style="text-align:left">While a t-test indicated that Excel and StarOffice Calc ratings for Task 6 were not significantly different (<i>t</i> (11) = -1.57; <i>p</i> > .05, n.s.), our qualitative observations during the usability tests suggest this task can be slightly easier in StarOffice Calc.?</p> |
| <p class=Picture style="</p> |
| <p style=" margin-left:.5in'><b>Explanation of the Problem</b></p> |
| <p class=MsoBodyText>Changing horizontal alignment of text in cells is virtually identical in Excel and StarOffice Calc; users simply click a toolbar button.? To align text <i>vertically</i>, however, Excel users must open the cell format menu, click on the alignment tab, and select a value from a vertical alignment drop-down field.? The <i>vertical alignment</i> menu is right next to <i>horizontal alignment</i>, and several users hesitated between the two; there aren?t any visual cues to let people quickly pick the right menu.?</p> |
| <p class=MsoBodyText>In StarOffice Calc, however, users have a virtually identical method available plus an additional accelerator: they need only click on convenient toolbar buttons to accomplish the same task.? The toolbar buttons are visual, depicting pictures of each alignment type, thus eliminating the need for the user to mentally process whether they want vertical or horizontal alignment.? Not all users found the toolbar button, but those who did indicated that its convenience pleased them.?</p> |
| <h5>Stage 2 - Adding Formulas</h5> |
| <p class=MsoBodyText>In Stage 2, participants were asked to add formulas to a financial spreadsheet, thus calculating sums, averages, and other functions across multiple rows and columns of numbers.? The stage consisted of two individual tasks, detailed in <span style='"REF _Ref532549633 \\r \\h"'>Figure 2</span> below.</p> |
| <p class=MsoBodyText>While the individual averages for each task are slightly higher in Excel, a t-test showed that overall, the differences between Excel and StarOffice Calc were not significant.??</p> |
| <p class=MsoBodyText>Based on <span style='"REF _Ref532549633 \\r \\h"'>Figure 2</span> and on our qualitative observations, we guessed that the difference between Excel and StarOffice Calc in Task 7, inserting a formula, might be significant.? Task 8 seemed fairly close, with users reporting StarOffice Calc to be about 5.6% more difficult than Excel, which is about equal to the difference we found in Stage 1.</p> |
| <br> |
| <img src="images/figure2.gif"><br> |
| <h6>Task 7: Inserting a formula</h6> |
| <p class="MsoBodyText">We performed a t-test and determined that there is no significant difference between Excel and StarOffice Calc for Task 7 (<i>t</i> (11) = 2.12; <i>p</i> > .05, n.s.). However, the distribution of individual scores for this task was bimodal in StarOffice Calc; it showed half the people giving it a 7 (very easy) and half evenly distributed between 2 and 5.</p> |
| <p style=" <i>p</i> > .05, n.s.).<span |
| style=""> </span>However, the distribution of individual scores |
| for this task was bimodal in StarOffice Calc; it showed half the people giving |
| it a 7 (very easy) and half evenly distributed between 2 and 5.<span |
| style=""> </span></p> |
| <p> </p> |
| <p class=MsoHeading7 style=" margin-top:0in;margin-right:0in;margin-bottom:0in; margin-left:.5in;margin-bottom:.0001pt;line-height:normal;'><b><span style="letter-spacing:-.25pt; |
| ">Explanation of the Problem</span></b></p> |
| <p class=MsoBodyText>The StarOffice Calc Function Autopilot is more complex than Excel?s Function dialog.? While many participants did not use the autopilot (they typed formulas in by hand), those who did often became confused.? The dialog, shown in <span style='"REF _Ref532629788 \\r \\h"'>Figure 3</span>, provides good explanations of the selected function in the main area; however, to pick the selected function, users have to first add it to the "Formula" box.? There is no "Add" button; they must know to double-click it.? Many users assumed they had selected the formula, clicked "OK," and then were surprised when the formula didn?t appear in the spreadsheet.</p> |
| <br> |
| <img src="images/figure3.gif"><br> |
| <p class=MsoBodyText>After selecting the desired formula, users were presented with the view shown in Figure 3.? Participants did not immediately understand what to put in each box; it took trial and error to create the correct function with the correct data range.</p> |
| <p class=MsoHeading7 style=" |
| line-height:normal;"><b><span style="letter-spacing:-.25pt; |
| ">Recommended Design Change</span></b></p> |
| <p class=MsoBodyText>StarOffice Calc needs a cleaner AutoPilot dialog box that is more intuitive to users.? A complete redesign of the box is out of scope of this study; the box should be designed iteratively and shown to users after each revision.? Also, it should be tested more comprehensively than we were able to, i.e. by having participants add multiple complex formulas to a document.</p> |
| <h5>Stage 3 - Advanced Formatting</h5> |
| <p class=MsoBodyText>In Stage 3, participants were asked to add advanced formatting to a financial spreadsheet.? The stage consisted of four individual tasks, detailed in <span style='"REF _Ref532550342 \\w"'>Figure 4</span> below.</p> |
| <p class=MsoBodyText>Tasks 9-11 follow the general pattern we have observed in the task data; StarOffice Calc tasks were ranked an average of 7.0% more difficult than Excel tasks, though a t-test indicated the overall difference between Excel and StarOffice Calc was not significant.</p> |
| <p class=MsoBodyText>We noticed that Task 12, drawing and formatting shapes, appears more difficult than the other tasks, especially in StarOffice Calc.? We chose to examine Task 12 in depth.</p> |
| <br> |
| <img src="images/figure4.gif"><br> |
| <h6>Task 12: Drawing and Formatting Shapes</h6> |
| <p class=MsoBodyText>A t-test indicated that user ratings were not significantly different between Excel and StarOffice Calc for creating and formatting shapes (<i>t</i> (11) = 1.46; <i>p</i> > .05, n.s.), but users had difficulty performing the task with both systems.?? One person failed to complete this task in Excel, and one failed to complete it in StarOffice Calc.? In StarOffice Calc, six people rated the task between 1 and 4 (Very Difficult through Neutral); in Excel, three people rated it between 1 and 4.?</p> |
| <p class=MsoHeading7 style=" |
| line-height:normal;"><b><span style="letter-spacing:-.25pt; |
| ">Explanation of the Problem</span></b></p> |
| <p class=MsoBodyText>Our participants struggled to complete this task for a few reasons.? First, of all the tasks we asked them to perform, it was the one they performed the least regularly in their normal use of spreadsheets.? Second, the drawing menus were difficult to locate in both systems.? Excel required the users to find the drawing toolbar located at the bottom of the screen, far away from the participants? locus of attention; StarOffice Calc required the participants to find the drawing icon in the small left navigation bar, figure out that it can be extended by clicking once and holding, and find the appropriate drawing tool.? The StarOffice Calc left menu is shown in <span style='"REF _Ref532631643 \\w"'>Figure 5</span>; the circle and arrow highlight the drawing menu icon.? People who failed to complete this task did so because they never did locate the drawing tools.</p> |
| <br> |
| <img src="images/figure5.gif"><br> |
| <p class=MsoBodyText>It should be noted that once participants found the drawing toolbar in StarOffice Calc, they did not have too many further problems in creating and coloring their shape.? They especially liked the way that text can be added to a shape simply by double-clicking (in Excel, users have to right-click and then choose an option from the menu).?</p> |
| <p class=MsoBodyText>They also found the shape color very easily.? The StarOffice Calc interface does an especially nice job of using people?s visual abilities to quickly indicate where the color can be changed.? <span style='"REF _Ref532632287 \\w"'>Figure 6</span> shows a blue circle that a user has just drawn; the color-change menu shows the same bright shade of blue, thus associating the shape and the color-change menu together in the user?s eye.?</p> |
| <p class=MsoBodyText>Despite these nicely-designed features, though, the time spent locating the shape menu in the first place seemed to influence people to rate the task as more difficult.</p> |
| <br> |
| <img src="images/figure6.gif"><br> |
| <p class=MsoHeading7 style=" |
| line-height:normal;"><b><span style="letter-spacing:-.25pt; |
| ">Recommended Design Change</span></b></p> |
| <p class=MsoBodyText>To improve the usability of the StarOffice Calc drawing feature, we recommend choosing a better icon for the left menu.? We also recommend adding a menu item called 'Insert Drawing' to the Insert menu, as users tended to look very hard for drawing tools in this area.?</p> |
| <h5>Stage 4 - Inserting a Chart</h5> |
| <p class=MsoBodyText>In Stage 4, participants were asked to use the spreadsheet data to create a simple bar chart.? The stage consisted of three individual tasks, detailed in <span style='"REF _Ref532552815 \\w"'>Figure 7</span> below.</p> |
| <p class=MsoBodyText>Overall, a t-test indicated that there was no significant difference between the systems in this stage. Tasks 14-15 follow the general pattern we have observed in the task data, having average StarOffice Calc rankings of 6.2% more than difficult than Excel ratings.?</p> |
| <p class=MsoBodyText>Task 13, inserting a chart, appears to be more difficult in StarOffice Calc than in Excel.? We chose to examine Task 13 in depth.? We also examine Task 14 because while it shows no significant difference between Excel and StarOffice Calc, one person in Excel and two people in StarOffice Calc failed to complete this task.</p> |
| <br> |
| <img src="images/figure7.gif"><br> |
| <h6>?Task 13: Inserting a Chart</h6> |
| <p class=MsoBodyText>The results indicate that participants' ease-of-use ratings for inserting a chart are significantly higher for Excel than for StarOffice Calc (<i>t</i> (11) = 2.26; <i>p</i> < .05).? In addition, two people failed to complete this task in StarOffice Calc.</p> |
| <p class=MsoBodyText><b>Explanation of the Problem</b></p> |
| <p class=MsoBodyText>Creating a chart is a multi-step process that many of our users don?t regularly complete. As such, they tended to run into problems both in Excel and StarOffice Calc. Generally, though, Calc charts took longer, caused more trouble, and were more likely to be incomplete or incorrect.</p> |
| <p class=MsoBodyText>In Calc, many of our participants:</p> |
| <ul> |
| <li>Don't know how to begin the chart process. <li>Do not notice the chart icon in left toolbar. When they do, they have difficulties realizing they have to click 'Chart' once, then again on the place in the spreadsheet where they want the chart to appear. |
| <li>Do not understand the 'Labels in first column' checkbox. |
| <li>Have a hard time altering the chart data range without starting over. |
| <li>Have difficulties selecting the appropriate data range (see Task 14). |
| <li>Do not understand 'data in rows or series' option. |
| </ul> |
| <p class=MsoBodyText>In general, participants found the entire process frustrating.? Some user comments were, "If it took me so long to do something, I would not do it.? It's going to drive me nuts," and "I like the Excel wizard better because it does a better job of predicting what I want."</p> |
| <p class=MsoBodyText><b>Recommended Design Change</b></p> |
| <p class=MsoBodyText>As with the function selection dialog, the chart dialog requires more thorough user testing and iterative design.? We had our users create a very basic chart, and requirements may be different for creating charts of different complexity or design.</p> |
| <p class=MsoBodyText>Based on our own study, we recommend the following changes:</p> |
| <ul> |
| <li>Place chart icon in a more obvious place.? Do not require complex clicking to create the chart; instead, have a single click open the chart dialog. |
| <li>Have a more visual way of indicating how to select a data range with labels.? Add a thumbnail diagram indicating what they are supposed to select. |
| <li>Make it easy to preview the whole chart.? Make it easier to re-do data selection without losing rest of chart formatting options. |
| <li>If people type in titles, put them in the chart automatically.? Do not require users to also click "include titles" checkboxes. |
| <li>Design a more visual, intuitive way to explain "rows/columns" distinction. |
| </ul> |
| <p> </p> |
| <h6>Task 14: Selecting Data Range</h6> |
| <p class="MsoBodyText">While our analysis indicates that there is no significant difference between Excel and StarOffice Calc for this task, two participants failed to complete this task correctly in StarOffice Calc, and one failed to complete it in Excel.</p> |
| <p class="MsoBodyText"><b>Explanation of the Problem</b></p> |
| <p class="MsoBodyText">We are unable to explain fully why some participants had difficulty with this step.? It seems that Excel and StarOffice Calc have taken similar approaches to support graph creation, but this approach is not terribly effective for many of our participants.?</p> |
| <p class="MsoBodyText"><b>Recommended Design Change</b></p> |
| <p class=MsoBodyText>If these systems are intended to support the creation of simple graphs by people who are not data experts, they need further work. We recommend further study of data selection and graph creation, examining different types of graphs, different data sets, and different user groups.</p> |
| <p> </p> |
| <h4>Comparative Time Analysis</h4> |
| <p class="MsoBodyText">Figure 8 displays the mean times taken to complete each stage.</p> |
| <br> |
| <img src="images/figure8.gif"><br> |
| <p class="MsoBodyText">The results of multivariate ANOVAs indicate that the differences in time between the two systems are only significant for Stage 4, chart creation (<i>F </i>(1,10) = 12.91; <i>p</i> < .01).? As reported above, chart creation is a weak point in the StarOffice Calc interface.</p> |
| <p>?</p> |
| <h4>System Ratings</h4> |
| <p class="MsoBodyText">This section reports participants? responses to questions administered directly after they finished testing each system.? These questions are not designed to be comparative, but rather to capture an initial reaction to each system.?</p> |
| <br> |
| <img src="images/figure9.gif"><br> |
| <p class="MsoBodyText">The graph shows that Excel scored higher than StarOffice Calc on all measures.? To test the significance of this observation, we performed paired-sample t-tests on all system comparison questions.? Because we conducted multiple t-tests, we chose to use the .01 significance level.? We found that differences were significant for half the items at this level.?? <span style='"REF _Ref532717427 \\r \\h"'>Table 9</span> shows the significant items in bold type:</p> |
| <p class=TableHeader><a name="_Ref532717427">Table 9.<span style='font:7.0pt "Times New Roman"'> </span>Differences between Excel and StarOffice Calc System Ratings</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=288 valign=top class="Normal"> </td> |
| <td width=283 valign=top class="Normal"> </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" ">1. Application is useful</p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" ">(<i>t</i> (11) = 3.74; <i>p</i> < .01)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">2. Application is easy to use</span></p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">(<i>t</i> (11) = 1.80; <i>p</i> > .01, n.s.)</span></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" ">3. Application is enjoyable</p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" ">(<i>t</i> (11) = 3.22; <i>p</i> < .01)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" ">4a. Can use without help</p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" ">(<i>t</i> (11) = 3.38; <i>p</i> < .01)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">4b. Can understand terminology</span></p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">(<i>t</i> (11) = 2.35; <i>p</i> > .01, n.s.)</span></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" ">4c. Can understand icons</p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" ">(<i>t</i> (11) = 3.25; <i>p</i> < .01)</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">4d. Can find things in menus</span></p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">(<i>t</i> (11) = 2.24; <i>p</i> > .01, n.s.)</span></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=288 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">4e. Like visual appearance</span></p> |
| </td> |
| <td width=283 valign=top class="Normal"> |
| <p class=TableHeader style=" "><span style="font-weight:normal;">(<i>t</i> (11) = 1.00; <i>p</i> > .01, n.s.)</span></p> |
| </td> |
| </tr> |
| </table> |
| <p><b>??????????????????????????????????? </b></p> |
| <p class="MsoBodyText">The results show that there were differences in participants? opinions about the usefulness of the two systems and their enjoyability.? The data also showed that participants felt that there were differences in their ability to understand the icons and to use the systems without help.? We believe that those four attributes are consistent with the behaviors of users who have had little exposure to an interface ? it seems logical that people wouldn?t enjoy a new interface as much as a familiar one, would need to use Help more often, and might not intuitively guess the meaning of unfamiliar icons.? We anticipate that with continued use of Calc, these ratings may improve.</p> |
| <h4>Predicting System Ease-of-Use</h4> |
| <p class="MsoBodyText">To determine what interface features contribute to users? overall opinions of a system, we conducted regressions to investigate whether each system?s Ease-of-Use rating can be predicted by each stage?s Average Task Ratings.</p> |
| <p class="MsoBodyText">In the Excel regression, only the Stage 3 (Advanced Formatting) Average Task Rating was included in the final model.? The regression equation was significant (<i>F</i> (1, 10) = 120.97; <i>p</i> < .01), and approximately 92% of the variance in ease-of-use can be attributed to Average Stage 3 Task Rating (adjusted <i>R</i><sup>2</sup> = .92).???</p> |
| <p class="MsoBodyText">We found that Calc Ease-of-Use can also be predicted by the StarOffice Calc Average Stage 3 Rating (<i>F</i> (1, 10) = 28.71; <i>p</i> < .01), and approximately 72% of the variance in ease-of-use can be attributed to Average Stage 3 Task Rating (adjusted <i>R</i><sup>2</sup> = .72).???</p> |
| <p class="MsoBodyText">Overall, this regression result is consistent with the qualitative data generated by the test.? Stage 3 tested advanced formatting skills such as drawing and formatting shapes, adding borders, adding background color, and changing font color.? Aside from cell merge, the steps required to complete Stages 1 and 2 were similar across the two systems, and Stage 4 was difficult for participants in both interfaces.? Stage 3, however, is where the steps needed to complete specific tasks differed significantly over a majority of the tasks.? Therefore, how easy or difficult a participant found the Stage 3 tasks tended to influence how the participant rated the system overall.</p> |
| <h4>Average System Satisfaction</h4> |
| <p class="MsoBodyText">The Average System Satisfaction is a variable combining the system-level ratings asked immediately after working with each system.? We found that the Average Excel Satisfaction was higher than the Average StarOffice Calc Satisfaction.? We found that the means were significantly different (<i>t</i> (11) = 5.04; <i>p</i> < .01) from one another.? The results are summarized in <span style='"REF _Ref532739116 \\n"'>Table 10</span> and <span style='"REF _Ref532739174 \\n"'>Figure 10</span>.</p> |
| <p class=TableHeader>Table 10.<span style='font:7.0pt "Times New Roman"'> </span>?????? <a name="_Ref532739116">Average System Satisfaction</a></p> |
| <table border=1 cellspacing=0 cellpadding=0> |
| <tr> |
| <td width=103 valign=top bgcolor="#E6E6E6" class="Normal"> </td> |
| <td width=102 valign=top bgcolor="#E6E6E6" class="Normal"> |
| <p><b>Mean</b></p> |
| </td> |
| <td width=144 valign=top bgcolor="#E6E6E6" class="Normal"> |
| <p><b>Standard Deviation</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=103 valign=top class="Normal"> |
| <p><b>Excel</b></p> |
| </td> |
| <td width=102 valign=top class="Normal"> |
| <p>5.91</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p>.64</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=103 valign=top class="Normal"> |
| <p><b>StarOffice Calc</b></p> |
| </td> |
| <td width=102 valign=top class="Normal"> |
| <p>4.93</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p>.83</p> |
| </td> |
| </tr> |
| </table> |
| <p>?</p> |
| <br> |
| <img src="images/figure10.gif"><br> |
| <p class="MsoBodyText">We expected the mean Excel satisfaction to be higher because our participants were more experienced with Excel.? However, Excel only scored 16% higher than StarOffice Calc on this metric, most of which we believe can be accounted for by the participants? lack of familiarity with StarCalc.</p> |
| <h4>Direct System Comparison and Evaluation</h4> |
| <p class="MsoBodyText">The direct system comparison questions were asked at the very end of the test, after participants had tested both systems.? We found significant correlations at the .01 level between the three comparison measures: Relative Usefulness, Relative Ease of Use, and Relative Enjoyment (see <span style='"REF _Ref532739525 \\n"'>Appendix F</span> for correlation matrix). We calculated the average scores of the three measures and found that Excel scored higher for all of them.</p> |
| <p class="MsoBodyText">We conducted a regression analysis to check whether the Average System Comparison score can predict Willingness to Switch to Calc. The regression equation was not significant, indicating that other factors might influence users? Willingness to Switch (Adjusted R? = .14, <i>F</i> (1, 10) = 2.81; p = .13, n.s.).</p> |
| <p class="MsoBodyText">The average score for satisfaction with Calc file compatibility with Excel was 4.63 (<i>s </i>= 1.85) on a 7-point scale. Note, however, that this score exhibited a bimodal distribution, with the two peaks at 3 and 6.? Given our observations about file compatibility during the tests, we had expected this rating to be more satisfactory than it is; we observed that files transferred almost perfectly with the exception of shapes and minor aspects of the graphs.? For a more detailed study of file compatibility between Microsoft and OpenOffice applications, we refer the reader to the Consulting Times study <a href="#reference3">[3]</a>.</p> |
| <h4>Would Participants Switch to StarOffice Calc?</h4> |
| <p class="MsoBodyText">Given the current price conditions of Excel (sold at $250 or more) and StarOffice Calc (available for free), 92% of our subjects indicated they would choose StarOffice Calc if they had to pick a spreadsheet system.? The distribution of responses is shown in <span style='"REF _Ref532739706 \\n"'>Table 11</span>:</p> |
| <p class=TableHeader><a name="_Ref532739706">Table 11.<span style='font:7.0pt "Times New Roman"'> </span>Willingness to Switch to StarOffice Calc</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=169 valign=top bgcolor="#E0E0E0" class="Normal"> </td> |
| <td width=108 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Frequency</b></p> |
| </td> |
| <td width=114 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Percent</b></p> |
| </td> |
| <td width=144 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Cumulative Percent</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p>Would always buy Excel</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p>1</p> |
| </td> |
| <td width=114 valign=top class="Normal"> |
| <p>8.3</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p>8.3</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p>Would buy Excel if the systems were equally priced, but would switch to StarOffice Calc if it were free</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p>5</p> |
| </td> |
| <td width=114 valign=top class="Normal"> |
| <p>41.7</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p>50</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p>Would buy Excel if the systems were equally priced, but would switch to Calc if it were $125 and Excel were $250</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p>5</p> |
| </td> |
| <td width=114 valign=top class="Normal"> |
| <p>41.7</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p class=TOCBase style="line-height: |
| normal;">91.7</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=169 valign=top class="Normal"> |
| <p>Would buy StarOffice Calc if the systems were equally priced, but would buy Excel if it were $125 and Calc were $250</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p>1</p> |
| </td> |
| <td width=114 valign=top class="Normal"> |
| <p>8.3</p> |
| </td> |
| <td width=144 valign=top class="Normal"> |
| <p>100</p> |
| </td> |
| </tr> |
| </table> |
| <p class="MsoBodyText">We see that StarOffice Calc works <i>well enough</i> that people?s decision to use it or not is generally dependent upon price.? Participants preferred Excel initially but were willing to switch if Excel cost them significantly more money.? When we asked participants about their responses to this question, the participants usually said they initially preferred Excel for two reasons.? First, the participant had already learned Excel.? Second, it would be easier for the participant to use the same system that friends and co-workers use.</p> |
| <h4>Effects of Participants? Previous Excel Experience</h4> |
| <p class="MsoBodyText">We began our study anticipating that certain people rating themselves with higher levels of experience would complete the test in shorter amounts of time.? However, our results show no significant correlations between any experience variables and any time variables, including individual stage times, Excel Total Time, StarOffice Calc Total Time, and Overall Total Time.? We also did not find any correlations between experience and how participants rated the two systems.?</p> |
| <h4> </h4> |
| <span style='font-size:14.0pt;font-family:"Arial Black"; |
| |
| letter-spacing:-.5pt;'><br clear=all style="page-break-before:always"> |
| </span> |
| <hr width="100%" size="1"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc532742119">Usability Problems and Design Recommendations</a></h2> |
| </div> |
| <hr width="100%" size="1"> |
| <p class="MsoBodyText">The following table summarizes the primary usability problems and suggested solutions that were explained in detail in the results section above.</p> |
| <p class=TableHeader>Table 12.<span style='font:7.0pt "Times New Roman"'> </span>Usability Problem Matrix</p> |
| <table border=1 cellspacing=0 cellpadding=4> |
| <tr> |
| <td width=122 valign=top bgcolor="#D9D9D9" class="Normal"> </td> |
| <td width=150 valign=top bgcolor="#D9D9D9" class="Normal"> |
| <p><b>Problem</b></p> |
| </td> |
| <td width=143 valign=top bgcolor="#D9D9D9" class="Normal"> |
| <p><b>User Comments</b></p> |
| </td> |
| <td width=159 valign=top bgcolor="#D9D9D9" class="Normal"> |
| <p class=MsoToc1><span style=" text-transform:none">Suggested Solution</span></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=122 valign=top class="Normal"> |
| <p><b>Finding and Applying Cell Merge</b></p> |
| </td> |
| <td width=150 valign=top class="Normal"> |
| <p>Cell merge is only available in Format menu.? Participants didn?t understand the term ?define.?</p> |
| <p>?</p> |
| <p>Participants didn?t understand that they had to unmerge already merged cells if they needed to add new cells to an already merged area.</p> |
| </td> |
| <td width=143 valign=top class="Normal"> |
| <p>?[Merge] must be in here somewhere.?</p> |
| <p>?What is this stupid define thing??</p> |
| <p>?You have to undo merge to redo it?? That?s stupid!?</p> |
| </td> |
| <td width=159 valign=top class="Normal"> |
| <p>Add a Merge shortcut button on the toolbar and in the Cell Attributes dialog box.</p> |
| <p>Allow users to merge new cells into an already merged area without unmerging the original merged area.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=122 valign=top class="Normal"> |
| <p><b>Drawing and Formatting Shapes</b></p> |
| </td> |
| <td width=150 valign=top class="Normal"> |
| <p>Function not available in pull-down menus.</p> |
| <p>Left navigation bar buttons are small and hard to manipulate. ?</p> |
| </td> |
| <td width=143 valign=top class="Normal"> |
| <p>?I'm totally lost here?</p> |
| </td> |
| <td width=159 valign=top class="Normal"> |
| <p>Add an Insert menu item called ?Insert Drawing? or add this option to the graphics menu item in the Insert menu.</p> |
| <p>Enlarge left hand button bar.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=122 valign=top class="Normal"> |
| <p><b>Inserting a chart</b></p> |
| </td> |
| <td width=150 valign=top class="Normal"> |
| <p>Can?t find insert chart function.</p> |
| <p>Left hand navigation bar buttons small and hard to manipulate.?</p> |
| <p>Hard time adjusting data range of chart w/o starting over.</p> |
| <p>Can?t type directly on chart to adjust titles or labels.</p> |
| <p>Comprehension difficulties with rows, series, columns.</p> |
| </td> |
| <td width=143 valign=top class="Normal"> |
| <p>?If it took me so long to do something, I would not do it.? It's going to drive me nuts.?</p> |
| <p>?This is why I don?t do these!? I hate these!?</p> |
| </td> |
| <td width=159 valign=top class="Normal"> |
| <p>Enlarge left hand navigation bar.</p> |
| <p>Find a more intuitive way to adjust data range for a chart.</p> |
| <p>Allow user to type directly on chart to change labels, titles, etc.</p> |
| <p>Conduct more comprehensive testing on chart fuctions.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=122 valign=top class="Normal"> |
| <p><b>Selecting Data Range for a Chart</b></p> |
| </td> |
| <td width=150 valign=top class="Normal"> |
| <p>Difficulty understanding which cell range to choose.</p> |
| </td> |
| <td width=143 valign=top class="Normal"> </td> |
| <td width=159 valign=top class="Normal"> |
| <p>Conduct more comprehensive testing on chart fuctions.</p> |
| </td> |
| </tr> |
| </table> |
| <h3><a name="_Toc532742120">Severity Metrics</a></h3> |
| <p class="MsoBodyText">We devised a four-point severity scale based on Jakob Nielsen?s severity ratings <a href="#reference2">[2]</a>.? The scale measures three main attributes of a usability problem: frequency, impact, and persistence.? Frequency refers to the percentage of participants affected by a given problem.? Impact refers to the ease with which a participant can overcome a given problem.? Persistence refers to how able participants are to overcome a given problem once they know about it.? Each usability problem receives a rating for each attribute.? The severity ratings are detailed in <span style='"REF _Ref532720737 \\r \\h"'>Table 13</span>.</p> |
| <p class=TableHeader><a name="_Ref532720737">Table 13.<span style='font:7.0pt "Times New Roman"'> </span>Severity Ratings</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=62 valign=top bgcolor="#E0E0E0" class="Normal"> </td> |
| <td width=137 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Frequency</b></p> |
| </td> |
| <td width=132 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Impact</b></p> |
| </td> |
| <td width=186 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Persistence</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=62 valign=top bgcolor="white" class="Normal"> |
| <p><b>0</b></p> |
| </td> |
| <td width=137 valign=top class="Normal"> |
| <p>Occurs very rarely</p> |
| </td> |
| <td width=132 valign=top class="Normal"> |
| <p>Minor annoyance</p> |
| </td> |
| <td width=186 valign=top class="Normal"> |
| <p>Problem does not recur after initial encounter.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=62 valign=top bgcolor="white" class="Normal"> |
| <p><b>1</b></p> |
| </td> |
| <td width=137 valign=top class="Normal"> |
| <p>Occurs in fewer than 10% of cases</p> |
| </td> |
| <td width=132 valign=top class="Normal"> |
| <p>Little impact</p> |
| </td> |
| <td width=186 valign=top class="Normal"> |
| <p>Problem can occur after initial encounter, but users generally are able to avoid problem.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=62 valign=top bgcolor="white" class="Normal"> |
| <p><b>2</b></p> |
| </td> |
| <td width=137 valign=top class="Normal"> |
| <p>Occurs in 10-50% of cases</p> |
| </td> |
| <td width=132 valign=top class="Normal"> |
| <p>Medium impact; takes some time to overcome</p> |
| </td> |
| <td width=186 valign=top class="Normal"> |
| <p>Users often re-commit error despite knowing about it.</p> |
| </td> |
| </tr> |
| <tr> |
| <td width=62 valign=top bgcolor="white" class="Normal"> |
| <p><b>3</b></p> |
| </td> |
| <td width=137 valign=top class="Normal"> |
| <p>Occurs in more than 50% of cases</p> |
| </td> |
| <td width=132 valign=top class="Normal"> |
| <p>Great impact; takes a disproportionate amount of time to overcome</p> |
| </td> |
| <td width=186 valign=top class="Normal"> |
| <p class="MsoBodyText">Users are unable to avoid encountering the problem even though they know it will occur.? The system will not allow them to bypass problem.</p> |
| </td> |
| </tr> |
| </table> |
| <p class="MsoBodyText"><span style='"REF _Ref532721409 \\w"'>Table 14</span> shows the four quantifiable usability problems discovered in the StarOffice Calc interface and our ratings of these problems using the abovementioned scale.</p> |
| <p class=TableHeader><a name="_Ref532721409">Table 14.<span style='font:7.0pt "Times New Roman"'> </span>StarOffice Calc Severity Ratings</a></p> |
| <table border=1 cellspacing=0 cellpadding=5> |
| <tr> |
| <td width=139 valign=top bgcolor="#E0E0E0" class="Normal"> </td> |
| <td width=96 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Frequency</b></p> |
| </td> |
| <td width=78 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Impact</b></p> |
| </td> |
| <td width=96 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Persistence</b></p> |
| </td> |
| <td width=108 valign=top bgcolor="#E0E0E0" class="Normal"> |
| <p><b>Total Severity Score</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=139 valign=top class="Normal"> |
| <p class=TOCBase style="line-height: |
| normal;">Finding and Applying Cell Merge</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>3</p> |
| </td> |
| <td width=78 valign=top class="Normal"> |
| <p>3</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>0</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p class=MsoToc1><span style=" text-transform:none">6</span></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=139 valign=top class="Normal"> |
| <p>Drawing and Formatting Shapes</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=78 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>0</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p><b>4</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=139 valign=top class="Normal"> |
| <p>Inserting a Chart</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=78 valign=top class="Normal"> |
| <p>3</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p><b>7</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=139 valign=top class="Normal"> |
| <p>Selecting Data Range</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=78 valign=top class="Normal"> |
| <p>3</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>2</p> |
| </td> |
| <td width=108 valign=top class="Normal"> |
| <p><b>7</b></p> |
| </td> |
| </tr> |
| <tr> |
| <td width=139 valign=top class="Normal"> |
| <p><b>Category Total</b></p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>9</p> |
| </td> |
| <td width=78 valign=top class="Normal"> |
| <p>11</p> |
| </td> |
| <td width=96 valign=top class="Normal"> |
| <p>4</p> |
| </td> |
| <td width=108 valign=top class="Normal"> </td> |
| </tr> |
| </table> |
| <h4>Suggestions for Developers</h4> |
| <p class=MsoBodyText align=left style="text-align:left"><span style='"REF _Ref532721409 \\w \\* MERGEFORMAT"'>Table 14</span> displays the severity ratings of the four major usability problems in the StarOffice Calc interface.? The following section offers additional comments based on our qualitative observations.</p> |
| <p class=MsoBodyText align=left style="text-align:left">The cell merge usability problem caused many complaints among our participants, although the problem did not persist after a participant's first encounter.? The cell merge problems are also the most discrete problems in the list.? Many of the other problems are interrelated and will require more re-design and user testing.? We are fairly confident that if developers allow users several paths to the merge function and allow users to merge cells into an already merged area, occurrences of cell merge errors will drop drastically.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Chart insertion and drawing and formatting shapes suffer from the same problem: participants' incomplete understanding of the left navigation bar.? The left navigation bar gave poor visual cues to our users in two areas: its size and its behavior.? It is perhaps understandable that participants accustomed to Excel would not immediately notice or use a left navigation bar.? However, the StarOffice Calc visual interface does not encourage new users to explore and comprehend this new tool.? The icons are so small that users have difficulty understanding them.? Increasing the size of the left navigation bar and its icons will help bring the bar and its functions to the attention of new users.? Applying this solution will help to reduce the frequency of the errors in chart insertion and shape creation.</p> |
| <p class=MsoBodyText align=left style="text-align:left">More damaging to the usability of StarOffice Calc was the left navigation bar's erratic behavior.? The buttons on the bar require the user to click and hold the mouse while a menu pops up; sometimes there is a significant delay between clicking and menu appearance.? Our participants occasionally explored the left navigation bar to see if it contained a function they needed.? When the participant clicked once on an icon, no action occurred.? This behavior is contradictory, not only to the menu behavior of Microsoft Office products, but also to the behavior of the menus at the top of Calc's interface.? Participants, focused on the task at hand, were likely to abandon the bar without viewing the bar's contents.? A good way to reduce the frequency and impact of some of the chart insertion and shape errors is to revise the look and behavior of the left hand navigation bar.</p> |
| <p class=MsoBodyText align=left style="text-align:left">Once participants discovered how to insert a chart in StarOffice Calc, they often had difficulty creating one.? We should note, however that Excel's chart interface also plagued our participants.? Our participants did not understand many of the terms (rows, series) or the conceptual model that the chart wizard process presented to them.? More testing should be done to understand how to improve chart interfaces, both in StarOffice Calc and Microsoft Excel.? We suggest implementing some basic fixes for the problems that some of participants encountered, including allowing users to revise titles, legends, and headings of charts by typing directly onto the chart.</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <h3><a name="_Toc532742121">Conclusions and Future Work</a></h3> |
| <p class="MsoBodyText">The primary objectives of this study were to discover usability problems in StarOffice Calc and to compare user satisfaction ratings between StarOffice Calc and Excel.? Our quantitative and qualitative data show that Excel users perform slightly better and are generally more satisfied with Excel than with StarOffice Calc.?</p> |
| <p class="MsoBodyText">Several problematic tasks emerged from our analysis.? Difficult tasks, in order of decreasing severity, are inserting a chart, selecting a data range for a chart, finding and applying cell merge, and drawing and formatting shapes.?</p> |
| <p class="MsoBodyText">However, several of these tasks were also difficult in Excel; this fact leads us to conclude that more research and user testing is needed to produce better-quality interfaces for spreadsheet products.? It should also be noted that the differences between Excel and Calc were generally small and not always significant.? In addition, participants commented that Calc behaved much better than they thought it would, and they were pleasantly surprised at its general overall usability.? We must also acknowledge that participants only had five minutes to acclimate themselves to StarOffice Calc before being asked to complete tasks.? We imagine that given a longer time to explore StarOffice Calc and real-life incentives to do so, people might grow to like it and perform as well as they do with Excel.</p> |
| <p class="MsoBodyText">We recognize that our study was limited; our participant pool was small and differed from the general population in many ways.? We were also able to test only a small subset of spreadsheet functions.? Future work should test a broader range of participants and a different set of tasks; StarOffice Calc might also benefit from a focused study of difficult tasks such as chart creation.? Such a study could then inform iterative design.</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc532742122">References</a></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <p class=MsoBodyText align=left style="text-align:left; |
| ">[1] <a name="reference1">Sun Corporate Website.</a>? "Desktop & Webtop, StarOffice 6.0 Beta Software."? Accessed 12/11/01,<br> |
| <strike> <a href="http://www.sun.com/software/star/staroffice/6.0beta/index.html">http://www.sun.com/software/star/staroffice/6.0beta/index.html</a>.</strike></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| ">[2] <a name="reference2">Nielsen, Jakob.</a>? "Severity Ratings for Usability Problems."? Accessed 12/11/01,<br> |
| <a href="http://www.useit.com/papers/heuristic/severityrating.html">http://www.useit.com/papers/heuristic/severityrating.html</a></p> |
| <p class=MsoBodyText align=left style="text-align:left; |
| ">[3] <a name="reference3">Zijleman, </a>Erwin and Harris, Stephen E. "The OpenOffice Rosetta Stone: Putting StarOffice/OpenOffice Conversions to the Test."?<br> |
| Accessed 12/11/01,<strike> <a href="http://consultingtimes.com/SOConversions.html">http://consultingtimes.com/SOConversions.html.</a></strike></p> |
| <p class=MsoBodyText>??????????????????????????????????????????????????????????????????</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <hr size="1" width="100%"> |
| <div style="border-top:solid white 6.0pt;border-left:solid white .75pt; |
| border-bottom:solid white .75pt;border-right:none;padding:3.0pt 0in 3.0pt 3.0pt"> |
| <h2 align="left"><a name="_Toc532742123">Appendices</a></h2> |
| </div> |
| <hr size="1" width="100%"> |
| <p class=AppendixStyle><a name="_Ref532735393"></a><a name="_Toc532742124">Appendix A.<span style='font:7.0pt "Times New Roman"'> </span>Questions</a></p> |
| <br> |
| <img src="images/questions1.gif"> <img src="images/questions2.gif"><br> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <p class="MsoBodyText">*Pricing question structure:</p> |
| <br> |
| <img src="images/tree.gif"><br> |
| <p> </p> |
| <p class=AppendixStyle><a name="_Ref532736936"></a><a name="_Toc532742125">Appendix B.<span style='font:7.0pt "Times New Roman"'> </span>Statement of Informed Consent</a></p> |
| <p class=MsoBodyText>A group of professors and graduate students in the School of Information Management and Systems, at UC Berkeley are conducting studies to assess the usability of some desktop software.</p> |
| <p class=MsoBodyText>If you volunteer to participate in this study, you will be asked to use one or more computer systems to ?use? some software designed for doing everyday tasks such as writing, doing calculations, painting etc.? and answer some questions about the software.? The questions will be asked orally before, and after your usage of the software.? There will also be written questionnaires before and after each activity.? Your interactions with the computer might be recorded on with keystroke logging software.</p> |
| <p class=MsoBodyText>There are no benefits to you for participating, other than what may be an educational experience in using software that you might not have used before.? We hope that the research will benefit society by improving the usability of software.? This research poses no risks to you other than those normally encountered in daily life. All of the information from your session will be kept confidential and be referred to by an ID number. The correspondence between your name and ID number will be kept confidential and treated with the same care as our own confidential information.? We will not name you if and when we discuss your behavior in research publications.? After the research is completed, we may save the notes for future use by ourselves or others.? However, this same confidentiality guarantees given here will apply to future storage and use of the materials.</p> |
| <p class=MsoBodyText>You will be paid $12 an hour for participation in this study. The study will take between one to two hours. Your participation in this research is voluntary, and you are free to refuse to participate or quit the experiment at any time.? Whether or not you chose to participate will have no bearing in relation to your standing in any department of UC Berkeley.? If you have questions about the research, you may contact</p> |
| <p class=MsoBodyText>Dr. Rashmi Sinha at 643-4314, or by electronic mail sinha@sims.berkeley.edu.? You may keep a copy of this form for reference.</p> |
| <p><i>Please sign and date below:</i></p> |
| <p><b>I have read this consent form and agree to participate in this research.</b></p> |
| <p><b>____________________________???? ___________________________ ___________</b></p> |
| <p><b>Signature?????????????????????????????????????? Full name??????????????????????????????? Date</b></p> |
| <b><span style='font-size:13.0pt;font-family:Arial;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <p class=AppendixStyle><a name="_Ref532737249"></a><a name="_Toc532742126">Appendix C.<span style='font:7.0pt "Times New Roman"'> </span>Instructions to Participants- Script</a></p> |
| <p class=MsoBodyTextIndent3>Note that this is the script for the group that does StarOffice Calc first. The script for the group that does Excel first is identical except for the order of the systems.</p> |
| <p class="MsoBodyText">Hi, my name is _____________.? I?ll be working with you in today?s session.? This is _________, and she'll be taking some notes as you work.? Let me explain why we?ve asked you to come in today.</p> |
| <p class="MsoBodyText">We?re here to test how easy it is to use spreadsheets to complete common tasks.? We are testing to see if there are differences between Excel and another spreadsheet product called StarOffice Calc.</p> |
| <p class="MsoBodyText">You will be performing some typical tasks with both spreadsheet programs today, and I?d like you to try and perform as you normally would.? For example, try to work at the same speed and with the same attention to detail that you normally do.? You may find that you have some difficulty finishing a particular task. Let us know at about the point you would normally abandon that task if you were using your normal spreadsheet program in real life-- and we'll move on. Do your best, but don?t worry too much about the results?we're testing the software, not you.? Also, be aware that one of our test programs is still in development, and it might not act as you expect.? There is a chance that the system will crash during your test, and if this happens, we'll let you know what to do.? You may ask questions at any time, but I may not answer them, since this is a study of the programs, and we need to see how well they work when someone uses them independently.?</p> |
| <p class="MsoBodyText">We will be timing your performance on these tasks, so don?t worry about thinking aloud, which is often done during usability tests.? Instead, concentrate on the task at hand, but do let me know if you have a comment or question, and we'll pause the timer.</p> |
| <p class="MsoBodyText">During today's session, we'll also be asking you to complete some forms and answer some questions.? It's important that you answer candidly.? Our role here today is to discover both the flaws and advantages of this product from your perspective; so don?t be shy about telling us what you think.</p> |
| <p class="MsoBodyText">Let?s begin by having you complete the consent form and the background information questionnaire.</p> |
| <p class="MsoBodyText"><i>(Administer Consent form and Background Questionnaire)</i></p> |
| <p class="MsoBodyText">You'll be working with two different spreadsheet programs today, StarOffice Calc and Excel.? we'll start you off in StarOffice Calc.? You'll complete a series of tasks for us and answer some questions.? Later, you'll complete a similar series of tasks in Excel.</p> |
| <p class="MsoBodyText">Today we?re going to use StarOffice Calc, a program you may not have used before. To get you started, you'll have 5 minutes to explore the system.? Take a look at the icons, menus, and toolbars.? After 5 minutes we'll start the test.</p> |
| <p class="MsoBodyText"><i>(System exploration - 5 minutes )</i></p> |
| <p class="MsoBodyText"><i>(Administrator opens StarOffice Calc file)</i></p> |
| <p class="MsoBodyText">Here?s the data file you'll be working with in StarOffice Calc</p> |
| <p class=MsoBodyTextIndent3>(Repeat the following for all four stages)</p> |
| <p class="MsoBodyText">We?d like you to change this data so that it looks like this</p> |
| <p class="MsoBodyText"><i>(show the printed sheet for StarOffice Calc for the relevant stage)</i></p> |
| <p class=Picture style='</p> |
| <p class=MsoBodyTextIndent3>(When they finish the relevant stage of StarOffice |
| Calc tasks)</p> |
| <p> </p> |
| <p class="MsoBodyText">Great.<span>? </span>Now I?m going |
| to ask you a few questions about what you just did.</p> |
| <p> </p> |
| <p class="MsoBodyText"><i>(ask task-level questions )</i></p> |
| <p><span>?</span></p> |
| <p class="MsoBodyText"><i>(When the user has completed the chart in StarOffice Calc)</i> |
| Great!<span>? </span>You?ve finished the first half |
| of the test!<span>? </span>We?d like you to save |
| this document in a way that you think will allow it to be opened in Excel.<span>? |
| </span>You can save it to the desktop...<span>? |
| </span>Now close the StarOffice Calc document ?</p> |
| <p> </p> |
| <p class="MsoBodyText">Now open your document with Excel.<span>? </span>Take a look and see if the data looks the same?<span>? </span>Now close the document ?</p> |
| <p> </p> |
| <p><i>(Ask StarOffice Calc general evaluation questions)</i></p> |
| <p> </p> |
| <p style=' margin-left:.25in;text-indent:.5in'><b><i>Switch to Excel</i></b></p> |
| <p class="MsoBodyText">This part of the test will be very similar to the last one.? I'm going to ask you to perform similar tasks on this Excel data set.?</p> |
| <p class=MsoBodyTextIndent3>(Repeat the following for all four stages)</p> |
| <br> |
| <br> |
| <p class=AppendixStyle><a name="_Ref532734650"></a><a name="_Ref532737299"></a><a name="_Toc532742127"><span>Appendix D.<span style='font:7.0pt "Times New Roman"'> </span>Screenshots of Stage Illustrations</span></a></p> |
| <h4>Excel Stage 1</h4> |
| <p align=center style="text-align:center">? <img width=575 height=504 src="images/image026.jpg" v:shapes="_x0000_i1043"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>Excel Stage 2</h4> |
| <p align=center style="text-align:center"><img width=576 height=504 src="images/image028.jpg" v:shapes="_x0000_i1044"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>Excel Stage 3</h4> |
| <p align=center style="text-align:center"><img width=576 height=504 src="images/image030.jpg" v:shapes="_x0000_i1045"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>Excel Stage 4</h4> |
| <p align=center style="text-align:center"><img width=576 height=504 src="images/image032.jpg" v:shapes="_x0000_i1046"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>StarOffice Calc Stage 1</h4> |
| <p align=center style="text-align:center"><img width=605 height=504 src="images/image034.jpg" v:shapes="_x0000_i1047"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>StarOffice Calc Stage 2</h4> |
| <p align=center style="text-align:center"><img width=610 height=504 src="images/image036.jpg" v:shapes="_x0000_i1048"></p> |
| <b><span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span></b> |
| <h4>StarOffice Calc Stage 3</h4> |
| <p align=center style="text-align:center"><img width=610 height=504 src="images/image038.jpg" v:shapes="_x0000_i1049"></p> |
| <span style='font-size:14.0pt;font-family:Garamond;"Times New Roman";'><br clear=all style="page-break-before:always"> |
| </span> |
| <h4>StarOffice Calc Stage 4</h4> |
| <p align=center style="text-align:center"><img width=612 height=504 src="images/image040.jpg" v:shapes="_x0000_i1050"></p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <p class=AppendixStyle><a name="_Ref532735101"></a><a name="_Ref532735161"></a><a name="_Ref532735470"></a><a name="_Ref532737170"></a><a name="_Toc532742128"><span><span>Appendix E.<span style='font:7.0pt "Times New Roman"'> </span>Question Scales</span></span></a></p> |
| <p class=MsoBodyTextIndent2>Experience:</p> |
| <ol> |
| <li>Never |
| <li>Yearly |
| <li>Monthly |
| <li>Weekly |
| <li>Daily |
| </ol> |
| <br> |
| <p class=MsoBodyTextIndent2>Satisfaction:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Very Unsatisfied??????????????????????????????????? ??????? Neutral????????????????????????????????????????? ???? Very Satisfied</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <br> |
| <p class=MsoBodyTextIndent2>Performance Expectation:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Much worse than Excel????????????? ?? No Difference????????????? ????????????????? Much better than Excel</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <br> |
| <p class=MsoBodyTextIndent2>Ease of use:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Very Difficult????????????????? ??? ??????????????????? ????????Neutral????????????????? ????????????????? ????? ???????????Very Easy</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <br> |
| <p class=MsoBodyTextIndent2>Usefulness:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Not at all Useful ??? ??????????????????? ????????Neutral????????????????? ????????????????? ????? ?????????Very Useful</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <br> |
| <p class=MsoBodyTextIndent2>Enjoyment:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Didn?t enjoy?????? ?? ???????? ??????????????????????? ????????Neutral????????????????? ?????? ???? ???????????Enjoyed very much</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <p class=MsoBodyTextIndent2>Agreement:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Strongly Disagree????????? ?? ???????? ??????????? ?????????Neutral???????????????? ?????? ???? ??????????????????Strongly Agree</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <br> |
| <p class=MsoBodyTextIndent2>Comparison:</p> |
| <p class=MsoBodyText style="border:none;padding:0in;">Excel??? ?? ???????? ??????????? ???????????They are the same????????????????????? ?????? ???? ????????????????????????????StarOffice Calc</p> |
| <hr align="left" size="1" width="70%"> |
| <br> |
| <p class=MsoBodyText>1????????????????????? 2????????????????????? 3????????????????????? 4????????????????????? 5????????????????????? 6????????? ??????????? 7?????????</p> |
| <span style="font-size:10.0pt;font-family:Arial; |
| letter-spacing:-.25pt;"><br clear=all style="page-break-before:always"> |
| </span> |
| <p class=AppendixStyle><a name="_Ref532739525"></a><a name="_Toc532742129">Appendix F.<span style='font:7.0pt "Times New Roman"'> </span>Correlations between Final System Comparison Questions</a></p> |
| <table border=1 cellspacing=0 cellpadding=5 width=256> |
| <tr> |
| <td width=64 nowrap valign=bottom bgcolor="#E6E6E6" class="Normal"> </td> |
| <td width=64 valign=top bgcolor="#E6E6E6" class="Normal"> |
| <p align=center style="text-align:center"><b>Useful</b><b></b></p> |
| </td> |
| <td width=64 valign=top bgcolor="#E6E6E6" class="Normal"> |
| <p align=center style="text-align:center"><b>Easy</b><b><span style='"Arial Unicode MS";'></span></b></p> |
| </td> |
| <td width=64 valign=top bgcolor="#E6E6E6" class="Normal"> |
| <p align=center style="text-align:center"><b>Enjoyable</b><b></b></p> |
| </td> |
| </tr> |
| <tr> |
| <td nowrap valign=bottom bgcolor="#E6E6E6" class="Normal"> |
| <p><b>Useful</b><b></b></p> |
| </td> |
| <td width=64 valign=top class="Normal"> |
| <p align=right style="text-align:right">1<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| <td nowrap valign=bottom class="Normal"> </td> |
| <td nowrap valign=bottom class="Normal"> </td> |
| </tr> |
| <tr> |
| <td nowrap valign=bottom bgcolor="#E6E6E6" class="Normal"> |
| <p><b>Easy</b><b></b></p> |
| </td> |
| <td nowrap valign=bottom class="Normal"> |
| <p align=right style="text-align:right">0.84**<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| <td nowrap valign=bottom class="Normal"> |
| <p align=right style="text-align:right">1<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| <td nowrap valign=bottom class="Normal"> </td> |
| </tr> |
| <tr> |
| <td nowrap valign=bottom bgcolor="#E6E6E6" class="Normal"> |
| <p><b>Enjoyable</b><b></b></p> |
| </td> |
| <td width=64 valign=top class="Normal"> |
| <p align=right style="text-align:right">0.74**<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| <td width=64 valign=top class="Normal"> |
| <p align=right style="text-align:right">0.81**<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| <td nowrap valign=bottom class="Normal"> |
| <p align=right style="text-align:right">1<span style='"Arial Unicode MS";'></span></p> |
| </td> |
| </tr> |
| </table> |
| <p class=MsoBodyText align=left style=" |
| text-align:left">??????????? **Significant at .01 level, two-tailed</p> |
| </td> |
| <td width="0%"></td> |
| </tr> |
| </table> |
| </body> |
| |
| </html> |
