서브메뉴
검색
Effective Differentially Private Deep Learning
Effective Differentially Private Deep Learning
상세정보
- 자료유형
- 학위논문 서양
- 최종처리일시
- 20250211152744
- ISBN
- 9798342113472
- DDC
- 006.35
- 저자명
- Li, Xuechen.
- 서명/저자
- Effective Differentially Private Deep Learning
- 발행사항
- [Sl] : Stanford University, 2024
- 발행사항
- Ann Arbor : ProQuest Dissertations & Theses, 2024
- 형태사항
- 154 p
- 주기사항
- Source: Dissertations Abstracts International, Volume: 86-04, Section: B.
- 주기사항
- Advisor: Guestrin, Carlos;Hashimoto, Tatsunori.
- 학위논문주기
- Thesis (Ph.D.)--Stanford University, 2024.
- 초록/해제
- 요약Deep learning models trained on sensitive data can leak privacy when deployed. For example, language models trained with standard algorithms can regurgitate training data and reveal membership information of data contributors. Differential Privacy (DP) is a formal guarantee that provably limits privacy leakage and has become the gold standard for privacy-preserving statistical data analysis. However, most approaches for training deep learning models with DP were computationally intensive and incurred substantial task performance penalties on the resulting model. This thesis presents improved techniques for deep learning with DP that are much more efficient and performant. These techniques have seen growing interest in the industry and have been used in differentially private machine learning deployments at major technology companies, protecting users' privacy and providing substantial computational savings.We show that Differentially Private Stochastic Gradient Descent (DP-SGD), when properly applied to fine-tune pretrained models of increasing size and quality, produces consistently better privacy-utility tradeoffs. DP-SGD is much more memory-intensive and slower compared to standard training algorithms. We present algorithmic and implementation modifications of DP-SGD, rendering it as efficient as standard training for Transformers models. Our empirical findings challenge the prevailing belief that DP-SGD performs poorly for optimizing high-dimensional objectives. To understand and explain our empirical results, we additionally present novel theoretical analyses on toy models that resemble large-scale fine-tuning and show that DP-SGD has dimension-independent bounds for a class of unconstrained convex optimization problems.
- 일반주제명
- Text categorization
- 일반주제명
- Deep learning
- 일반주제명
- Fines & penalties
- 일반주제명
- Information processing
- 일반주제명
- Privacy
- 기타저자
- Stanford University.
- 기본자료저록
- Dissertations Abstracts International. 86-04B.
- 전자적 위치 및 접속
- 로그인 후 원문을 볼 수 있습니다.
MARC
008250123s2024 us c eng d■001000017163719
■00520250211152744
■006m o d
■007cr#unu||||||||
■020 ▼a9798342113472
■035 ▼a(MiAaPQ)AAI31520270
■035 ▼a(MiAaPQ)Stanfordfw148my0453
■040 ▼aMiAaPQ▼cMiAaPQ
■0820 ▼a006.35
■1001 ▼aLi, Xuechen.
■24510▼aEffective Differentially Private Deep Learning
■260 ▼a[Sl]▼bStanford University▼c2024
■260 1▼aAnn Arbor▼bProQuest Dissertations & Theses▼c2024
■300 ▼a154 p
■500 ▼aSource: Dissertations Abstracts International, Volume: 86-04, Section: B.
■500 ▼aAdvisor: Guestrin, Carlos;Hashimoto, Tatsunori.
■5021 ▼aThesis (Ph.D.)--Stanford University, 2024.
■520 ▼aDeep learning models trained on sensitive data can leak privacy when deployed. For example, language models trained with standard algorithms can regurgitate training data and reveal membership information of data contributors. Differential Privacy (DP) is a formal guarantee that provably limits privacy leakage and has become the gold standard for privacy-preserving statistical data analysis. However, most approaches for training deep learning models with DP were computationally intensive and incurred substantial task performance penalties on the resulting model. This thesis presents improved techniques for deep learning with DP that are much more efficient and performant. These techniques have seen growing interest in the industry and have been used in differentially private machine learning deployments at major technology companies, protecting users' privacy and providing substantial computational savings.We show that Differentially Private Stochastic Gradient Descent (DP-SGD), when properly applied to fine-tune pretrained models of increasing size and quality, produces consistently better privacy-utility tradeoffs. DP-SGD is much more memory-intensive and slower compared to standard training algorithms. We present algorithmic and implementation modifications of DP-SGD, rendering it as efficient as standard training for Transformers models. Our empirical findings challenge the prevailing belief that DP-SGD performs poorly for optimizing high-dimensional objectives. To understand and explain our empirical results, we additionally present novel theoretical analyses on toy models that resemble large-scale fine-tuning and show that DP-SGD has dimension-independent bounds for a class of unconstrained convex optimization problems.
■590 ▼aSchool code: 0212.
■650 4▼aText categorization
■650 4▼aDeep learning
■650 4▼aFines & penalties
■650 4▼aInformation processing
■650 4▼aPrivacy
■690 ▼a0800
■71020▼aStanford University.
■7730 ▼tDissertations Abstracts International▼g86-04B.
■790 ▼a0212
■791 ▼aPh.D.
■792 ▼a2024
■793 ▼aEnglish
■85640▼uhttp://www.riss.kr/pdu/ddodLink.do?id=T17163719▼nKERIS▼z이 자료의 원문은 한국교육학술정보원에서 제공합니다.
