Mahmut Çağdaş DURMAZOĞLU, Leman Figen GÜL

Exploring usability tests to evaluate designers’ interaction with mobile augmented reality application for conceptual architectural design

Architectural design could be defined as a process in which the information about various aspects of a design object is produced at different stages, then shared between relevant stakeholders. For the early stages of the architectural design, in order to carry out an efficient and productive design process, and create comprehensive solutions to design problems, novel digital tools have to be developed. Digital become omnipresent in the contemporary architectural practice. Even though very complex design ideas could not be realized without the help of the Computer-aided Design (CAD) tools, some may argue that the idea generation is hindered by the employment of them during the early phases. With the idea of containing the essence of analogue tools, we propose an Augmented Reality (AR) application for architects to explore 3D mass geometries in a similar immediacy and ease of designing with a pen-paper. In this paper, we present and discus the validity of two chosen usability scales System Usability Scale (SUS) and Handheld Augmented Reality Usability Scale (HARUS) measuring comprehensibility of the developed AR application (MimAR). The results of the preliminary study shows that the chosen measurement methods provide a similar tendency of scores. The findings of this study suggests that the comprehensibility of the MimAR appli cation still needs improvement, and yet the usability of the application could be considered above the acceptable threshold.

PDF

___

Akin, Ö., & Weinel, E. F. (1982). Representation and architecture. Information Dynamics Inc.
Aliakseyeu, D., Martens, J.B., & Rauterberg, M. (2006). A computer support tool for the early stages of architectural design. Interacting with Computers, 18(4), 528-555. https://doi. org/10.1016/j.intcom.2005.11.010
Azuma, R. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4), 355-385. https://doi.org/10.1162/ pres.1997.6.4.355
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6), 34-47. https:// doi.org/10.1109/38.963459
Bai, H., Gao, L., El-Sana, J., & Billinghurst, M. (2013). Markerless 3D gesture-based interaction for handheld augmented reality interfaces. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 1-6). IEEE. https://doi. org/10.1109/ISMAR.2013.6671841
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114-123.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574-594. https:// doi.org/10.1080/10447310802205776
Boeykens, S., Santana Quintero, M., & Neuckermans, H. (2008). Improving architectural design analysis using 3D modeling and visualization techniques. In Digital Heritage: Proceedings of the 14th International Conference on Virtual Systems and Multi media (pp. 67-73). Archeolingua.
Brooke, J. (1996). SUS-A quick and dirty usability scale. In P. W. Jordan, B. Thomas, I. L. McClelland, B. Weerdmeeester (Eds.), Usability evaluation in industry (pp. 4-7). CRC Press. https:// doi.org/10.1201/9781498710411
Brooke, J. (2013). SUS: a retrospective. Journal of Usability Studies, 8(2), 29-40.
Caudell, T. P., & Mizell, D. W. (1992). Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences. Kauai, HI, USA. https:// doi.org/10.1109/HICSS.1992.183317
Chen, Z. R. (2007). How to improve creativity. In A. Dong, A. V. Moere, & G. S. Gero (Eds.), Proceedings of Computer-Aided Architectural Design Futures (CAADFutures) (pp. 571-583). Springer. https://doi.org/10.1007/978- 1-4020-6528-6
Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In J.J. O’Hare (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 213-218). Association for Computing Machinery. https://doi. org/10.1145/57167.57203
Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.
Cross, N. (2011). Design thinking: Understanding how designers think and work. Berg.
Dünser, A., & Billinghurst, M. (2011). Evaluating augmented reality systems. In B. Furht (Ed.), Handbook of augmented reality (pp. 289-307). Springer. https://doi.org/10.1007/978- 1-4614-0064-6_13
Ens, B., Quigley, A., Yeo, H. S., Irani, P., Piumsomboon, T., & Billinghurst, M. (2017, November 27-30). Exploring mixed-scale gesture interaction [Conference Poster]. SIGGRAPH Asia 2017 Posters, Bangkok, Thailand. https://doi. org/10.1145/3145690.3145740
Funk, M., Kritzler, M., & Michahelles, F. (2017, October 22-25). HoloLens is more than air tap: Natural and intuitive interaction with holograms [Conference Poster]. In IOT’17: Proceedings of the Seventh International Conference on the Internet of Things, Linz, Austria. https:// doi.org/10.1145/3131542.3140267
Goel, V. (1995). Sketches of thought. MIT Press.
Goldschmidt, G. (1991). The dialectics of sketching. Creativity Research Journal, 4(2), 123-143. https://doi. org/10.1080/10400419109534381
Goldschmidt, G. (2004). Design representation: Private process, public image. In G. Goldschmidt, & W. L. Porter (Eds.), Design representation (pp. 203-217). Springer. https://doi. org/10.1007/978-1-85233-863-3_9
Gross, M. D., & Do, E. Y.-L. (1996). Ambiguous intentions: a paper-like interface for creative de sign. In D. Kurlander, M. Brown, & R. Rao (Eds.), UIST’96: Proceedings of the 9th annual ACM symposium on User interface software and technology (pp. 183-192). Association for Computing Machinery. https://doi. org/10.1145/237091.237119
Gruenefeld, U., Ennenga, D., Ali, A. E., Heuten, W., & Boll, S. (2017). EyeSee360: Designing a visualization technique for out-of-view objects in head-mounted augmented reality. In SUI’17: Proceedings of the 5th Symposium on Spatial User Interaction (pp.109-118). Association for Computing Machinery. https:// doi.org/10.1145/3131277.3132175
Gül, L. F. (2008). Affordances of virtual environments: Do design media change the interaction with the design representation? In W. Nakapan, E. Mahaek, K. Teeraparbwong, & P. Nilkaew (Eds.), 13th International Conference on Computer Aided Architectural Design Research in Asia, CAADRIA 2008, Beyond Computer-Aided Design (pp. 213-220). Pimniyom Press
Gül, L. F. (2017). Studying architectural massing strategies in co-de sign-mobile augmented reality tool versus 3D virtual world. In A. Fioravanti , S. Cursi, S. Elahmar, S. Gargaro, G. Loffreda, G. Novembri, A. Trento (Eds.), ShoCK! - Sharing Computational Knowledge! - Proceedings of the 35th eCAADe Conference (pp. 703-710). Faculty of Civil and Industrial Engineering, Sapienza University of Rome.
Gül, L. F. (2018). Studying gesture-based interaction on a mobile aug mented reality application for co-design activity. Journal on Multimodal User Interfaces, 12(2), 109-124. https://doi. org/10.1007/s12193-017-0252-0
Hegde, S., Perla, R., Hebbalaguppe, R., & Hassan, E. (2016). GestAR: Real time gesture interaction for AR with egocentric view. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) (pp. 262-267). IEEE. https://doi.org/10.1109/ ISMAR-Adjunct.2016.0090
Henrysson, A., Marshall, J., & Billinghurst, M. (2007). Experiments in 3D interaction for mobile phone AR. In GRAPHITE’07: Proceedings of the 5th international conference on Computer graphics and interactive techniques in Australia and Southeast Asia (pp. 187-194). Association for Computing Machinery. https://doi. org/10.1145/1321261.1321295
Hürst, W., & van Wezel, C. (2011). Multimodal interaction concepts for mobile augmented reality applica¬tions. In K. T. Lee, W. H. Tsai, H. Y. M. Liao, T. Chen, J. W. Hsieh, & C. C. Tseng (Eds.), Advances in Multimedia Modeling: 17th International Multi¬media Modeling Conference MMM 2011 (pp. 157-167). Springer. https://doi.org/10.1007/978- 3-642-17829-0_15
Hürst, W., & Van Wezel, C. (2013). Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools and Applications, 62(1), 233-258. https://doi. org/10.1007/s11042-011-0983-y
Ibrahim, R., & Rahimian, F. P. (2010). Comparison of CAD and manual sketching tools for teaching architectural design. Automation in Construction, 19(8), 978-987. https://doi. org/10.1016/j.autcon.2010.09.003
Kalay, Y. E. (2006). The impact of in formation technology on design meth ods, products and practices. Design Studies, 27(3), 357-380. https://doi. org/10.1016/j.destud.2005.11.001
Kirakowski, J., & Corbett, M. (1993). SUMI: the software usability measurement inventory. British Journal of Educational Technology, 24(3), 210-212. https://doi. org/10.1111/j.1467-8535.1993. tb00076.x
Knight, M., Dokonal, W., Brown, A., & Hannibal, C. (2005). Contemporary digital techniques in the early stages of design. In B. Martens, & A. Brown (Eds.), Computer Aided Architectural Design Futures 2005 (pp. 165-174). Springer. https://doi.org/10.1007/1- 4020-3698-1_15
Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human–Computer Interaction, 7(1), 57-78. https:// doi.org/10.1080/10447319509526110
Lewis, J. R., & Sauro, J. (2018). Item benchmarks for the system usability scale. Journal of Usability studies, 13(3). 158-167
Lin, C.-Y. (2001). A digital procedure of building construction: a practical project. In K. Ny, P. Tom, J. Verbeke, & J. Verleye (Eds.), AVOCAAD - Added value of computer aided architectural design (pp. 459-468). Hogeschool voor Wetenschap en Kunst - Departement Architectuur Sint-Lucas.
Mitchell, W. J., & McCullough, M. (1995). Digital design media (2nd ed.). Wiley & Sons.
Nielsen, J. (2012, June 3). How many test users in a usability study? Nielsen Norman Group. https://www.nngroup. com/articles/how-many-test-users/
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In CHI’93: Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (pp. 206-213). Association for Computing Machinery. https:// doi.org/10.1145/169059.169166
Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction design: Beyond human-computer interaction (4th ed.). Wiley.
Rahimian, F. P., & Ibrahim, R. (2011). Impacts of VR 3D sketching on novice designers’ spatial cognition in collabrative conceptual architectural design. Design Studies, 32(3), 255-291. https:// doi.org/10.1016/j.destud.2010.10.003
Reffat, R. (2007). Revitalizing architectural design studio teaching using ICT: Reflections on practical im-plementations. International Journal of Education and Development us¬ing ICT, 3(1), 39-53.
Ryu, Y. S., & Smith-Jackson, T. L. (2006). Reliability and validity of the mobile phone usability ques¬tionnaire (MPUQ). Journal of Usability Studies, 2(1), 39-53.
Sachse, P., Leinert, S., & Hacker, W. (2001). Designing with computer and sketches. Swiss Journal of Psychology / Schweizerische Zeitschrift für Psycholo gie / Revue Suisse de Psychologie, 60(2), 65-72. https://doi.org/10.1024/1421- 0185.60.2.65
Salman, H., Laing, R., & Conniff, A. (2008). The changing role of CAAD in the architectural design studio. The Built and Human Environment Re view, 1, 25-39.
Santos, M., Ericson, C. , Polvi, J., Taketomi,T., Yamamoto, G., Sandor, C., & Kato, H. (2015). Toward standard usability questionnaires for handheld augmented reality. IEEE Computer Graphics and Applications, 35(5), 66-75. https://doi.org/10.1109/MCG.2015.94
Santos, M., Ericson, C., Taketomi, T., Sandor, C., Polvi, J., Yamamoto, G., & Kato, H. (2014). A usability scale for handheld augmented reality. In VRST’14: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology (pp. 167-176), Association for Computing Machinery. https:// doi.org/10.1145/2671015.2671019
Schon, D. A. (2008). The reflective practitioner: How professionals think in action. Hachette.
Shih, Y. T., Sher, W. D., & Taylor, M. (2017). Using suitable design media appropriately: Understanding how designers interact with sketching and CAD modelling in design processes. Design Studies, 53, 47-77. https://doi. org/10.1016/j.destud.2017.06.005
Simon, H. A. (1995). Problem forming, problem finding and problem solving in design. In A.Collen, & W. W. Gasparski (Eds.), Design & Systems: General applications of methodology (Volume 3 of Praxiology: International annual of practical philosophy and methodology series, pp. 245-257). Transaction Publishers.
Simon, H. A. (1996). The sciences of the artificial (3rd ed.). MIT Press. Stones, C., & Cassidy, T. (2007). Comparing synthesis strategies of novice graphic designers using digital and traditional design tools. Design Studies, 28(1), 59-72. https://doi. org/10.1016/j.destud.2006.09.001
Sutherland, I. E. (1968). A head-mounted three dimensional display. In AFIPS’68 (Fall, Part I) Proceedings of the Fall Joint Computer Conference, part I (pp. 757-764). Association for Computing Machinery. https:// doi.org/10.1145/1476589.1476686
Suwa, M., Purcell, T., & Gero, J. S. (1998). Macroscopic analysis of design processes based on a scheme for coding designers’ cognitive actions. Design Studies, 19(4), 455- 483. https://doi.org/10.1016/S0142- 694X(98)00016-7
Tang, H. H., Lee, Y. Y., & Gero, J. S. (2011). Comparing collaborative co-located and distributed design processes in digital and traditional sketching environments: A protocol study using the function–behaviour–structurecoding scheme. Design Studies, 32(1),1-29. https://doi. org/10.1016/j.destud.2010.06.004
Wang, X. (2009). Augmented reality in architecture and design: Potentials and challenges for application. International Journal of Architectural Computing, 7(2), 309-326. https://doi. org/10.1260/147807709788921985
Won, P. H. (2001). The comparison between visual thinking using computer and conventional media in the concept generation stages of design. Automation in Construction, 10(3), 319-325. https:// doi.org/10.1016/S0926-5805(00)00048-0
Xue, H., Sharma, P., & Wild, F. (2019). User satisfaction in augmented reality-based training using Microsoft HoloLens. Computers, 8(1), 1-23. https://doi. org/10.3390/computers8010009
Zimmer, C., Bertram, M., Büntig, F., Drochtert, D., & Geiger, C. (2017, November 27-30). Mobile augmented reality illustrations that entertain and inform: Design and implementation issues with the HoloLens [Conference presentation abstract]. SA’17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications, Bangkok, Thailand. https://doi. org/10.1145/3132787.3132804