서브메뉴
검색
Leveraging Sparsity in Deep Neural Networks for Training Efficiency, Interpretability and Generalization
Leveraging Sparsity in Deep Neural Networks for Training Efficiency, Interpretability and Generalization
상세정보
- 자료유형
- 학위논문 서양
- 최종처리일시
- 20260202105600
- ISBN
- 9798263399337
- DDC
- 006.312
- 서명/저자
- Leveraging Sparsity in Deep Neural Networks for Training Efficiency, Interpretability and Generalization
- 발행사항
- [Sl] : Georgia Institute of Technology, 2024
- 발행사항
- Ann Arbor : ProQuest Dissertations & Theses, 2024
- 형태사항
- 196 p
- 주기사항
- Source: Dissertations Abstracts International, Volume: 87-05, Section: B.
- 주기사항
- Advisor: Dovrolis, Constantine.
- 학위논문주기
- Thesis (Ph.D.)--Georgia Institute of Technology, 2024.
- 초록/해제
- 요약Sparse neural networks (Sparse NNs) are characterized by having fewer connectionsbetween consecutive layers compared to traditional fully connected, or dense NNs. Historically, sparsity has been studied post-training to enhance inference efficiency and as aregularization mechanism to improve generalization. However, additional benefits beyondthese areas remain underexplored. In this thesis, we investigate sparse NNs, various sparsity patterns, and their broader benefits, including improved training efficiency, enhancedinterpretability, and stronger generalization.First, we introduce PHEW (Path with Higher Edge-Weights), a novel method for identifying sparse sub-networks within dense NNs at initialization, without using any trainingdata. PHEW is a probabilistic network formation method based on biased random walks,relying solely on the initial weights of the NN. Importantly, PHEW does not make anytask-specific assumptions; instead, it exploits structural properties inherent in dense NNsthat promote faster convergence and better generalization. By identifying effective sparsesub-networks at initialization, PHEW reduces the computational burden of training denseNNs and consistently outperforms other state-of-the-art methods.Second, we propose Neural Sculpting, a technique to uncover the underlying hierarchicallymodular task structure within NNs. Many real-world tasks exhibit hierarchical modularity,where complex target functions can be decomposed into simpler sub-functions arrangedin a hierarchy. We pose the following question: given a sufficiently deep NN, how canwe uncover the task's hierarchical structure? Neural Sculpting uses an iterative process ofpruning both units and edges during training, followed by network analysis to detect functional modules and infer hierarchical relationships between them. This method enhancesthe interpretability of NNs by guiding them to reflect the task's inherent hierarchical andmodular structure through pruning, and subsequently revealing that structure through network analysis.Finally, we leverage structural information about the task's hierarchical modularity to enhance NN performance by aligning the architecture at initialization with the task's structure. Specifically, we investigate how modular NNs can outperform dense NNs by systematically varying the degree of structural knowledge incorporated at initialization. Wecompare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hierarchically modular NNs with shared modules, which leverage sparsity, modularity, and module reusability. Incorporating modularity and module reuse significantly enhances learning efficiency and generalization, particularly in data-scarce scenarios, wherehierarchically modular NNs excel by promoting functional specialization and reducing redundancy. These findings reveal that task-specific architectural biases can lead to moreefficient, interpretable, and effective learning systems.In conclusion, this thesis demonstrates that sparse NNs offer not only enhanced trainingand inference efficiency but also superior interpretability and generalization capabilities.These findings have broad implications for NN design across various domains, particularlyin data-scarce scenarios or applications where understanding the underlying task structureis essential. Future work may focus on refining these methodologies and extending theirapplicability to more complex, real-world tasks and larger-scale architectures.
- 일반주제명
- Clustering
- 일반주제명
- Success
- 일반주제명
- Graphs
- 일반주제명
- Boolean
- 일반주제명
- Visualization
- 일반주제명
- Neural networks
- 기본자료저록
- Dissertations Abstracts International. 87-05B.
- 전자적 위치 및 접속
- 로그인 후 원문을 볼 수 있습니다.
MARC
008260126s2024 us c eng d■001000017360643
■00520260202105600
■006m o d
■007cr#unu||||||||
■020 ▼a9798263399337
■035 ▼a(MiAaPQ)AAI32315954
■035 ▼a(MiAaPQ)GeorgiaTech76897
■040 ▼aMiAaPQ▼cMiAaPQ
■0820 ▼a006.312
■1001 ▼aPatil, Shreyas Malakarjun .
■24510▼aLeveraging Sparsity in Deep Neural Networks for Training Efficiency, Interpretability and Generalization
■260 ▼a[Sl]▼bGeorgia Institute of Technology▼c2024
■260 1▼aAnn Arbor▼bProQuest Dissertations & Theses▼c2024
■300 ▼a196 p
■500 ▼aSource: Dissertations Abstracts International, Volume: 87-05, Section: B.
■500 ▼aAdvisor: Dovrolis, Constantine.
■5021 ▼aThesis (Ph.D.)--Georgia Institute of Technology, 2024.
■520 ▼aSparse neural networks (Sparse NNs) are characterized by having fewer connectionsbetween consecutive layers compared to traditional fully connected, or dense NNs. Historically, sparsity has been studied post-training to enhance inference efficiency and as aregularization mechanism to improve generalization. However, additional benefits beyondthese areas remain underexplored. In this thesis, we investigate sparse NNs, various sparsity patterns, and their broader benefits, including improved training efficiency, enhancedinterpretability, and stronger generalization.First, we introduce PHEW (Path with Higher Edge-Weights), a novel method for identifying sparse sub-networks within dense NNs at initialization, without using any trainingdata. PHEW is a probabilistic network formation method based on biased random walks,relying solely on the initial weights of the NN. Importantly, PHEW does not make anytask-specific assumptions; instead, it exploits structural properties inherent in dense NNsthat promote faster convergence and better generalization. By identifying effective sparsesub-networks at initialization, PHEW reduces the computational burden of training denseNNs and consistently outperforms other state-of-the-art methods.Second, we propose Neural Sculpting, a technique to uncover the underlying hierarchicallymodular task structure within NNs. Many real-world tasks exhibit hierarchical modularity,where complex target functions can be decomposed into simpler sub-functions arrangedin a hierarchy. We pose the following question: given a sufficiently deep NN, how canwe uncover the task's hierarchical structure? Neural Sculpting uses an iterative process ofpruning both units and edges during training, followed by network analysis to detect functional modules and infer hierarchical relationships between them. This method enhancesthe interpretability of NNs by guiding them to reflect the task's inherent hierarchical andmodular structure through pruning, and subsequently revealing that structure through network analysis.Finally, we leverage structural information about the task's hierarchical modularity to enhance NN performance by aligning the architecture at initialization with the task's structure. Specifically, we investigate how modular NNs can outperform dense NNs by systematically varying the degree of structural knowledge incorporated at initialization. Wecompare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hierarchically modular NNs with shared modules, which leverage sparsity, modularity, and module reusability. Incorporating modularity and module reuse significantly enhances learning efficiency and generalization, particularly in data-scarce scenarios, wherehierarchically modular NNs excel by promoting functional specialization and reducing redundancy. These findings reveal that task-specific architectural biases can lead to moreefficient, interpretable, and effective learning systems.In conclusion, this thesis demonstrates that sparse NNs offer not only enhanced trainingand inference efficiency but also superior interpretability and generalization capabilities.These findings have broad implications for NN design across various domains, particularlyin data-scarce scenarios or applications where understanding the underlying task structureis essential. Future work may focus on refining these methodologies and extending theirapplicability to more complex, real-world tasks and larger-scale architectures.
■590 ▼aSchool code: 0078.
■650 4▼aClustering
■650 4▼aSuccess
■650 4▼aGraphs
■650 4▼aBoolean
■650 4▼aVisualization
■650 4▼aNeural networks
■690 ▼a0800
■71020▼aGeorgia Institute of Technology.
■7730 ▼tDissertations Abstracts International▼g87-05B.
■790 ▼a0078
■791 ▼aPh.D.
■792 ▼a2024
■793 ▼aEnglish
■85640▼uhttp://www.riss.kr/pdu/ddodLink.do?id=T17360643▼nKERIS▼z이 자료의 원문은 한국교육학술정보원에서 제공합니다.
