서브메뉴
검색
Improving Web Automation Tools Through UI Context and Demonstration
Improving Web Automation Tools Through UI Context and Demonstration
상세정보
- 자료유형
- 학위논문 서양
- 최종처리일시
- 20250211152104
- ISBN
- 9798382740683
- DDC
- 004
- 서명/저자
- Improving Web Automation Tools Through UI Context and Demonstration
- 발행사항
- [Sl] : University of Michigan, 2024
- 발행사항
- Ann Arbor : ProQuest Dissertations & Theses, 2024
- 형태사항
- 126 p
- 주기사항
- Source: Dissertations Abstracts International, Volume: 85-12, Section: B.
- 주기사항
- Advisor: Oney, Steve.
- 학위논문주기
- Thesis (Ph.D.)--University of Michigan, 2024.
- 초록/해제
- 요약User interface (UI) automation allows people to perform UI tasks programmatically and can be helpful for computer or smartphone tasks that are tedious, repetitive, or inaccessible. UI automation works by programmatically mimicking a user's interactions on a UI, for example clicking a button or typing into a text field. Traditionally people create UI automation macros by writing code, which requires programming expertise and familiarity with UI technologies. Researchers have explored direct manipulation interfaces and programming-by-demonstration (PBD) to make creating UI automation more accessible for people with less programming experience. With PBD, the user provides demonstrations of how they want their program to behave in a small set of scenarios, and the system then infers a generalized program. Since demonstrations are inherently ambiguous, a key challenge of PBD is in correctly inferring the user's intent and effectively communicating those inferences back to the user. In this thesis, I address important challenges in authoring UI automation macros by leveraging user-provided demonstrations and parameters, and structural patterns in the UI to infer generalized automation; and in understanding UI automation macros by (a) highlighting selected elements on the target UI, (b) visualizing high-level behavior through sequences of actions and UIs visited, (c) visualizing generalizations through color-coding UI elements and grouping corresponding UIs, and (d) providing feedback on validity and uniqueness of element selection logic. First, I conducted two studies observing how programmers write automation code. One of the key challenges participants experienced was in identifying appropriate UI element selection logic. Next, I designed two programming-by-demonstration systems, ParamMacros and ScrapeViz, that enable users to create automation macros without writing code. Users provide demonstrations of what UI elements they want to click or scrape, and then these systems leverage structural patterns in the website DOM to identify patterns and infer generalized automation. ParamMacros supports parameterized macros (powered by user-provided parameters) while ScrapeViz supports distributed hierarchical web scraping macros. ScrapeViz also provides visual tools to help users understand automation behavior in the context of the page source and across different UI pages. This thesis contributes learnings about the challenges users face in creating UI automation macros, and no-code authoring tools and visual understanding tools which have the promise to make UI automation more accessible to a wider audience.
- 일반주제명
- Computer science
- 일반주제명
- Information science
- 일반주제명
- Computer engineering
- 일반주제명
- Information technology
- 키워드
- Web automation
- 키워드
- User interfaces
- 기타저자
- University of Michigan Computer Science & Engineering
- 기본자료저록
- Dissertations Abstracts International. 85-12B.
- 전자적 위치 및 접속
- 로그인 후 원문을 볼 수 있습니다.
MARC
008250123s2024 us c eng d■001000017162858
■00520250211152104
■006m o d
■007cr#unu||||||||
■020 ▼a9798382740683
■035 ▼a(MiAaPQ)AAI31349058
■035 ▼a(MiAaPQ)umichrackham005502
■040 ▼aMiAaPQ▼cMiAaPQ
■0820 ▼a004
■1001 ▼aKrosnick, Rebecca P.
■24510▼aImproving Web Automation Tools Through UI Context and Demonstration
■260 ▼a[Sl]▼bUniversity of Michigan▼c2024
■260 1▼aAnn Arbor▼bProQuest Dissertations & Theses▼c2024
■300 ▼a126 p
■500 ▼aSource: Dissertations Abstracts International, Volume: 85-12, Section: B.
■500 ▼aAdvisor: Oney, Steve.
■5021 ▼aThesis (Ph.D.)--University of Michigan, 2024.
■520 ▼aUser interface (UI) automation allows people to perform UI tasks programmatically and can be helpful for computer or smartphone tasks that are tedious, repetitive, or inaccessible. UI automation works by programmatically mimicking a user's interactions on a UI, for example clicking a button or typing into a text field. Traditionally people create UI automation macros by writing code, which requires programming expertise and familiarity with UI technologies. Researchers have explored direct manipulation interfaces and programming-by-demonstration (PBD) to make creating UI automation more accessible for people with less programming experience. With PBD, the user provides demonstrations of how they want their program to behave in a small set of scenarios, and the system then infers a generalized program. Since demonstrations are inherently ambiguous, a key challenge of PBD is in correctly inferring the user's intent and effectively communicating those inferences back to the user. In this thesis, I address important challenges in authoring UI automation macros by leveraging user-provided demonstrations and parameters, and structural patterns in the UI to infer generalized automation; and in understanding UI automation macros by (a) highlighting selected elements on the target UI, (b) visualizing high-level behavior through sequences of actions and UIs visited, (c) visualizing generalizations through color-coding UI elements and grouping corresponding UIs, and (d) providing feedback on validity and uniqueness of element selection logic. First, I conducted two studies observing how programmers write automation code. One of the key challenges participants experienced was in identifying appropriate UI element selection logic. Next, I designed two programming-by-demonstration systems, ParamMacros and ScrapeViz, that enable users to create automation macros without writing code. Users provide demonstrations of what UI elements they want to click or scrape, and then these systems leverage structural patterns in the website DOM to identify patterns and infer generalized automation. ParamMacros supports parameterized macros (powered by user-provided parameters) while ScrapeViz supports distributed hierarchical web scraping macros. ScrapeViz also provides visual tools to help users understand automation behavior in the context of the page source and across different UI pages. This thesis contributes learnings about the challenges users face in creating UI automation macros, and no-code authoring tools and visual understanding tools which have the promise to make UI automation more accessible to a wider audience.
■590 ▼aSchool code: 0127.
■650 4▼aComputer science
■650 4▼aInformation science
■650 4▼aComputer engineering
■650 4▼aInformation technology
■653 ▼aWeb automation
■653 ▼aProgramming by demonstration
■653 ▼aUser interfaces
■653 ▼aEnd-user programming
■653 ▼aHuman-computer interaction
■690 ▼a0984
■690 ▼a0723
■690 ▼a0489
■690 ▼a0464
■71020▼aUniversity of Michigan▼bComputer Science & Engineering.
■7730 ▼tDissertations Abstracts International▼g85-12B.
■790 ▼a0127
■791 ▼aPh.D.
■792 ▼a2024
■793 ▼aEnglish
■85640▼uhttp://www.riss.kr/pdu/ddodLink.do?id=T17162858▼nKERIS▼z이 자료의 원문은 한국교육학술정보원에서 제공합니다.
