Deep neural networks in hydrology: the new generation of universal and efficient models
DOI:
https://doi.org/10.21638/spbu07.2021.101Abstract
For around a decade, deep learning - the sub-field of machine learning that refers to artificial neural networks comprised of many computational layers - has been modifying the landscape of statistical model development in many research areas, such as image classification, machine translation, and speech recognition. Geoscientific disciplines in general and the field of hydrology in particular, are no exception to this movement. Recently, the proliferation of modern deep learning-based techniques and methods has been actively gaining popularity for solving a wide range of hydrological problems: modeling and forecasting of river runoff, hydrological model parameters regionalization, assessment of available water resources, and identification of the main drivers of the recent change in water balance components. This growing popularity of deep neural networks is primarily due to their high universality and efficiency. The presented qualities, together with the rapidly growing amount of accumulated environmental information, as well as the increasing availability of computing facilities and resources, allow us to speak about deep neural networks as a new generation of mathematical models designed to, if not to replace existing solutions, then significantly enrich the field of geophysical processes modeling. This paper provides a brief overview of the current state of the field of development and application of deep neural networks in hydrology. Also in the following study, the qualitative long-term forecast regarding the development of deep learning technology for managing the corresponding hydrological modeling challenges is provided based on the use of the Gartner Hype Curve, which in the general details describes the life cycle of modern technologies.
Keywords:
deep neural networks, deep learning, machine learning, hydrology, modeling
Downloads
References
Addor, N., Newman, A. J., Mizukami, N. and Clark, M. P. (2017). The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences (HESS), 21 (10), 5293-5313.
Adiwardana, D., Luong, M. T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y. and Le, Q. V. (2020). Towards a Human-like Open-Domain Chatbot. ArXiv, [online] Available at: https://arxiv.org/abs/2001.09977 [Accessed Mar. 04, 2021].
Agrawal, S., Barrington, L., Bromberg, C., Burge, J., Gazen, C. and Hickey, J. (2019). Machine Learning for Precipitation Nowcasting from Radar Images. ArXiv, [online] Available at: https://arxiv.org/ abs/1912.12132 [Accessed Mar. 04, 2021].
Arazo, E., Ortego, D., Albert, P., O’Connor, N. E. and McGuinness, K. (2019). Pseudo-labeling and confirmation bias in deep semi-supervised learning. ArXiv, [online] Available at: https://arxiv.org/ abs/1908.02983 [Accessed Mar. 04, 2021].
Arsenault, R., Bazile, R., Ouellet Dallaire, C. and Brissette, F. (2016). CANOPEX: A Canadian hydrometeorological watershed database. Hydrological Processes, 30 (15), 2734-2736.
Avanzi, F., Johnson, R. C., Oroza, C. A., Hirashima, H., Maurer, T. and Yamaguchi, S. (2020). Insights into preferential-flow snowpack runoff using Random Forest. Water Resources Research, 55 (12), 10727- 10746.
Ayzel, G. (2019). Does Deep Learning Advance Hourly Runoff Predictions? In: Proceedings of the V International Conference on Information Technologies and High-Performance Computing. [online] Khabarovsk: CEUR-WS, 62-70. Available at: http://ceur-ws.org/Vol-2426/paper9.pdf [Accessed Mar. 04, 2021].
Ayzel, G., Heistermann, M., Sorokin, A., Nikitin, O. and Lukyanova, O. (2019). All convolutional neural networks for radar-based precipitation nowcasting. Procedia Computer Science, 150, 186-192.
Ayzel, G., Kurochkina, L., Kazakov, E. and Zhuravlev, S. (2020). Runoff prediction in ungauged basins: benchmarking the efficiency of deep learning. In: IV Vinogradov Conference “Hydrology: from Learning to Worldview” in Memory of Outstanding Russian Hydrologist Yury Vinogradov. [online] Saint Peters- burg: E3S Web Conf., 1-6. Available at: https://doi.org/10.1051/e3sconf/202016301001 [Accessed Mar. 04, 2021].
Bauer, P., Thorpe, A. and Brunet, G. (2015). The quiet revolution of numerical weather prediction. Nature, 525 (7567), 47-55.
Bremer, L. L., Hamel, P., Ponette-González, A. G., Pompeu, P. V., Saad, S. I. and Brauman, K. A. (2019). Who are we measuring and modeling for? Supporting multi-level decision-making in watershed management. Water Resources Research, 56 (1), e2019WR026011.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K. and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on computer vision and pattern recognition (CVPR2009). [online] Miami: IEEE, 248-255. Available at: https://ieeexplore.ieee.org/document/5206848 [Accessed Mar. 04, 2021].
Fuhrer, O., Chadha, T., Hoefler, T., Kwasniewski, G., Lapillonne, X., Leutwyler, D., Lüthi, D., Osuna, C., Schär, C., Schulthess, T. C. and Vogt, H. (2018). Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0. Geoscientific Model Development, 11 (4), 1665-1681.
Gauch, M., Mai, J. and Lin, J. (2019). The Proper Care and Feeding of CAMELS: How Limited Training Data Affects Streamflow Prediction. ArXiv, [online] Available at: https://arxiv.org/abs/1911.07249 [Accessed Mar. 04, 2021].
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S. and Yang, G. Z. (2019). XAI - Explainable artificial intelligence. Science Robotics, 4 (37), eaay7120.
Ho, T. K. (1995). Random decision forests. In: Proceedings of 3rd international conference on document analysis and recognition. [online] Montreal: IEEE, 1, 278-282 [Accessed Mar. 04, 2021].
Hornik, K., Stinchcombe, M. and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2 (5), 359-366.
Hinton, G. E., Osindero, S. and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural computation, 18 (7), 1527-1554.
Hutter, F., Kotthoff, L. and Vanschoren, J. (2019). Automated Machine Learning. New York, USA: Springer.
Konapala, G. and Mishra, A. (2020). Quantifying climate and catchment control on hydrological drought in continental United States. Water Resources Research, 56 (1), e2018WR024620.
Kratzert, F., Klotz, D., Brenner, C., Schulz, K. and Herrnegger, M. (2018). Rainfall-runoff modelling using long short-term memory (LSTM) networks. Hydrology and Earth System Sciences, 22 (11), 6005-6022.
Kratzert, F., Herrnegger, M., Klotz, D., Hochreiter, S. and Klambauer, G. (2019a). NeuralHydrology- Interpreting LSTMs in Hydrology. In: W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, K.-R. Müller, ed., Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Bern: Springer, 347-362.
Kratzert, F., Klotz, D., Herrnegger, M., Sampson, A. K., Hochreiter, S. and Nearing, G. S. (2019b). Toward Improved Predictions in Ungauged Basins: Exploiting the Power of Machine Learning. Water Resources Research, 55 (12), 11344-11354.
Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems 25 (NIPS 2012). [online] Lake Tahoe: NIPS, 1097-1105. Available at: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep- convolutional-neural-networks.pdf [Accessed Mar. 04, 2021].
Laloy, E., Hérault, R., Jacques, D. and Linde, N. (2018). Training-image based geostatistical inversion using a spatial generative adversarial neural network. Water Resources Research, 54 (1), 381-406.
LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning. Nature, 521 (7553), 436-444.
McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5 (4), 115-133.
McGovern, A., Lagerquist, R., John Gagne, D., Jergensen, G. E., Elmore, K. L., Homeyer, C. R. and Smith, T. (2019). Making the black box more transparent: Understanding the physical implications of machine learning. Bulletin of the American Meteorological Society, 100 (11), 2175-2199.
Minsky, M. and Papert, S. A. (1969). Perceptrons: An introduction to computational geometry. Boston: MIT Press.
Moraux, A., Dewitte, S., Cornelis, B. and Munteanu, A. (2019). Deep Learning for Precipitation Estimation from Satellite and Rain Gauges Measurements. Remote Sensing, 11 (21), 2463.
Olah, C., Mordvintsev, A. and Schubert, L. (2017). Feature visualization. Distill, 2 (11), e7.
O’Leary, D. E. (2008). Gartner’s hype cycle and information system research issues. International Journal of Accounting Information Systems, 9 (4), 240-252.
Pan, B., Hsu, K., AghaKouchak, A. and Sorooshian, S. (2019). Improving precipitation estimation using convolutional neural network. Water Resources Research, 55 (3), 2301-2321.
Racah, E., Beckham, C., Maharaj, T., Kahou, S. E., Prabhat, M. and Pal, C. (2017). Extreme Weather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. In: Advances in Neural Information Processing Systems 31 (NIPS 2017). [online] Long Beach: NIPS, 3402-3413. Available at: https://papers.nips.cc/paper/6932-extremeweather-a-large- scale-climate-dataset-for-semi-supervised-detection-localization-and-understanding-of-extreme- weather-events.pdf [Accessed Mar. 04, 2021].
Ranzato, M. A., Poultney, C., Chopra, S. and LeCun, Ya. (2007). Efficient learning of sparse representations with an energy-based model. In: Advances in neural information processing systems 21 (NIPS 2007). [online] Vancouver: NIPS, 1137-1144. Available at: https://papers.nips.cc/paper/3112-efficient- learning-of-sparse-representations-with-an-energy-based-model.pdf [Accessed Mar. 04, 2021].
Rao, Q. and Frtunikj, J. (2018). Deep learning for self-driving cars: chances and challenges. In: Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems. [online] Gothenburg: IEEE, 35-38. Available at: https://ieeexplore.ieee.org/document/8452728 [Accessed Mar. 04, 2021].
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J. and Carvalhais, N. (2019). Deep learning and process understanding for data-driven Earth system science. Nature, 566 (7743), 195-204.
Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65 (6), 386.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1 (5), 206-215.
Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1985). Learning internal representations by error propagation. [report] California Univ San Diego La Jolla, Inst for Cognitive Science, San Diego.
Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323 (6088), 533-536.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. and Berg, A. C. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 115 (3), 211-252.
Samek, W., Wiegand, T. and Müller, K. R. (2017). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. [online] Available at: https://arxiv.org/abs/1708.08296 [Accessed Mar. 04, 2021].
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. [online] Venice: IEEE, 618-626. Available at: https://ieeexplore.ieee.org/document/8237336 [Accessed Mar. 04, 2021].
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural networks, 61, 85-117.
Shen, C. (2018). A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resources Research, 54 (11), 8558-8593.
Shen, C., Laloy, E., Elshorbagy, A., Albert, A., Bales, J., Chang, F. J., Ganguly, S., Hsu, K. L., Kifer, D., Fang, Z. and Fang, K. (2018). HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community. Hydrology and Earth System Sciences, 22 (11), 5639-5656.
Shi, X., Gao, Z., Lausen, L., Wang, H., Yeung, D. Y., Wong, W. K. and Woo, W. C. (2017). Deep learning for precipitation nowcasting: A benchmark and a new model. In: Advances in neural information processing systems 30 (NIPS 2017). [online] Long Beach: NIPS, 5617-5627. Available at: https://papers.nips.cc/paper/7145-deep-learning-for-precipitation-nowcasting-a-benchmark-and-a-new-model [Accessed Mar. 04, 2021].
Simonyan, K., Vedaldi, A. and Zisserman, A. (2014). Deep inside convolutional networks: Visualising image classification models and saliency maps. [online] Available at: https://arxiv.org/abs/1312.6034 [Accessed Mar. 04, 2021].
Song, X., Zhang, G., Liu, F., Li, D., Zhao, Yu. and Yang, J. (2016). Modeling spatio-temporal distribution of soil moisture by deep learning-based cellular automata model. Journal of Arid Land, 8 (5), 734-748.
Song, T., Ding, W., Wu, J., Liu, H., Zhou, H. and Chu, J. (2020). Flash Flood Forecasting Based on Long Short-Term Memory Networks. Water, 12 (1), 109.
Sun, C., Shrivastava, A., Singh, S. and Gupta, A. (2017). Revisiting unreasonable effectiveness of data in deep learning era. In: Proceedings of the IEEE international conference on computer vision. [online] Venice: IEEE, 843-852. Available at: https://ieeexplore.ieee.org/document/8237359 [Accessed Mar. 04, 2021].
Sutskever, I., Vinyals, O. and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In: Advances in neural information processing systems 27 (NIPS 2014). [online] Montreal: NIPS, 3104- 3112. Available at: https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neuralnetworks.pdf [Accessed Mar. 04, 2021].
Tao, Yu., Gao, X., Hsu, K., Sorooshian, S. and Ihler, A. (2016). A deep neural network modeling framework to reduce bias in satellite precipitation products. Journal of Hydrometeorology, 17 (3), 931-945.
Trends.google.com. (2020). Google Trends’ Official Website. [online] Available at: https://trends.google.com [Accessed Mar. 04, 2021].
Wagstaff, K. L. and Lee, J. (2018). Interpretable discovery in large image data sets. [online] Available at: https://arxiv.org/pdf/1806.08340.pdf [Accessed Mar. 04, 2021].
Webofscience.com. (2020). Web of Science Core Collection’s Official Website. [online] Available at: https://apps.webofknowledge.com [Accessed Mar. 04, 2021].
Zhang, L., Zhang, L. and Du, B. (2016). Deep learning for remote sensing data: A technical tutorial on the state of the art. IEEE Geoscience and Remote Sensing Magazine, 4 (2), 22-40.
Zhang, J., Liu, P., Zhang, F. and Song, Q. (2018). CloudNet: Ground-based cloud classification with deep convolutional neural network. Geophysical Research Letters, 45 (16), 8665-8672.
Zhao, W. and Du, S. (2016). Learning multiscale and deep representations for classifying remotely sensed imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 113, 155-165.
Zoph, B., Vasudevan, V., Shlens, J. and Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. [online] Salt Lake City: IEEE, 8697-8710. Available at: https://ieeexplore.ieee.org/document/8579005 [Accessed Mar. 04, 2021].
Downloads
Published
How to Cite
Issue
Section
License
Articles of "Vestnik of Saint Petersburg University. Earth Sciences" are open access distributed under the terms of the License Agreement with Saint Petersburg State University, which permits to the authors unrestricted distribution and self-archiving free of charge.
