| Title: | Prunability of multi-layer perceptrons trained with the forward-forward algorithm |
|---|
| Authors: | ID Nikov, Mitko (Author)
ID Strnad, Damjan (Author)
ID Podgorelec, David (Author) |
|---|
| Files: |  mathematics-13-02668-v2.pdf (7,01 MB)
MD5: C04BE66BFE440B30671ECBB8D216ACF6
|
|---|
| Language: | English |
|---|
| Work type: | Article |
|---|
| Typology: | 1.01 - Original Scientific Article |
|---|
| Organization: | FERI - Faculty of Electrical Engineering and Computer Science
|
|---|
| Abstract: | We explore the sparsity and prunability of multi-layer perceptrons (MLPs) trained using the Forward-Forward (FF) algorithm, an alternative to backpropagation (BP) that replaces the backward pass with local, contrastive updates at each layer. We analyze the sparsity of the weight matrices during training using multiple metrics, and test the prunability of FF networks on the MNIST, FashionMNIST and CIFAR-10 datasets. We also propose FFLib—a novel, modular PyTorch-based library for developing, training and analyzing FF models along with a suite of FF-based architectures, including FFNN, FFNN+C and FFRNN. In addition to structural sparsity, we describe and apply a new method for visualizing the functional sparsity of neural activations across different architectures using the HSV color space. Moreover, we conduct a sensitivity analysis to assess the impact of hyperparameters on model performance and sparsity. Finally, we perform pruning experiments, showing that simple FF-based MLPs exhibit significantly greater robustness to one-shot neuron pruning than traditional BP-trained networks, and a possible 8-fold increase in compression ratios while maintaining comparable accuracy on the MNIST dataset. |
|---|
| Keywords: | Forward-Forward, sparsity, pruning, model compression, machine learning, neural network |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Submitted for review: | 22.06.2025 |
|---|
| Article acceptance date: | 15.08.2025 |
|---|
| Publication date: | 19.08.2025 |
|---|
| Publisher: | MDPI |
|---|
| Year of publishing: | 2025 |
|---|
| Number of pages: | 23 str. |
|---|
| Numbering: | Vol. 13, iss. 16, [article no.] 2668 |
|---|
| PID: | 20.500.12556/DKUM-94526  |
|---|
| UDC: | 004.9 |
|---|
| ISSN on article: | 2227-7390 |
|---|
| COBISS.SI-ID: | 246013187 |
|---|
| DOI: | 10.3390/math13162668 |
|---|
| Copyright: | © 2025 by the authors |
|---|
| Publication date in DKUM: | 20.08.2025 |
|---|
| Views: | 153 |
|---|
| Downloads: | 8 |
|---|
| Metadata: |    |
|---|
| Categories: | Misc.
|
|---|
|
:
|
Copy citation |
|---|
| | | | Average score: | (0 votes) |
|---|
| Your score: | Voting is allowed only for logged in users. |
|---|
| Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
