Journal of Data Science and Modeling

On weight and variance uncertainty in neural networks for regression tasks

Document Type : Research Manuscript

Authors

Morteza Amini 1
Moein Monemi 2
Mahmoud Taheri 2
Mohammad Arashi 3

1 Department of Statistics, School of Mathematics, Statistics Aand Computer Science, College of Science, University of Tehran

2 School of Engineering Science, College of Engineering, University of Tehran, Tehran, Iran

3 Department of Statistics, Faculty of Mathematical Sciences, Ferdowsi University of Mashhad, P.O. Box 1159, Mashhad 91775, Iran

https://doi.org/10.22054/jdsm.2026.88820.1080
Abstract
We investigate the problem of weight uncertainty originally proposed by [Blundell et al. (2015). Weight uncertainty in neural networks. In International conference on machine learning, 1613-1622, PMLR.] in the context of neural networks designed for regression tasks, and we extend their framework by incorporating variance uncertainty into the model. Our analysis demonstrates that explicitly modeling uncertainty in the variance parameter can significantly enhance the predictive performance of Bayesian neural networks. By considering a full posterior distribution over the variance, the model achieves improved generalization compared to approaches that treat variance as fixed or deterministic. We evaluate the generalization capability of our proposed approach through a function approximation example and further validate it on the riboflavin genetic dataset. Our exploration encompasses both fully connected dense networks and dropout neural networks, employing Gaussian and spike-and-slab priors respectively for the network weights, providing a comprehensive assessment of how variance uncertainty affects model performance across different architectural choices.

Keywords

Bayesian neural networks
Variational Bayes
Kullback-Leibler divergence
posterior distribution
Regression task
variance uncertainty
Subjects
Journal of Data Science and Modeling
Volume 3, Issue 1 - Serial Number 5
December 2024
Pages 201-222

Home

Guide for Authors

Submit Manuscript

Contact Us