Levent Sagun (EPF Lausanne)
Theory of Deep Learning 2: Over-parametrization in neural networks: an overview and a definition abstract
Abstract:
An excursion around the ideas for why the stochastic gradient descent algorithm works well on training deep neural networks leads to considerations about the underlying geometry of the related loss function. Recently, we gained a lot of insight into how tuning SGD leads to better or worse generalization properties on a given model and task. Furthermore, we have a reasonably large set of observations that lead to the conclusion that more parameters typically lead to better accuracies as long as the training process is not hampered. In this talk, I will speculatively argue that as long as the model is over-parameterized (OP), all solutions are equivalent up to finite size fluctuations. We will start by reviewing some of the recent literature on the geometry of the loss function, and how SGD navigates the landscape in the OP regime. Then we will see how to define OP by finding a sharp transition described by the models fitting abilities to its training set. Finally, we will discuss how this critical threshold is connected to the generalization properties of the model, and argue that life beyond this threshold is (more or less) as good as it gets.
11:00 • Universität Basel, Fachbereich Mathematik, Spiegelgasse 1, Room 05.002
Levent Sagun (EPFL)
Abstract:
An excursion around the ideas for why the stochastic gradient descent algorithm works well on training deep neural networks leads to considerations about the underlying geometry of the related loss function. Recently, we gained a lot of insight into how tuning SGD leads to better or worse generalization properties on a given model and task. Furthermore, we have a reasonably large set of observations that lead to the conclusion that more parameters typically lead to better accuracies as long as the training process is not hampered. In this talk, I will speculatively argue that as long as the model is over-parameterized (OP), all solutions are equivalent up to finite size fluctuations. We will start by reviewing some of the recent literature on the geometry of the loss function, and how SGD navigates the landscape in the OP regime. Then we will see how to define OP by finding a sharp transition described by the models fitting abilities to its training set. Finally, we will discuss how this critical threshold is connected to the generalization properties of the model, and argue that life beyond this threshold is (more or less) as good as it gets.
11:00 • Universität Basel
Manfred K. Warmuth (UC Santa Cruz and Google Zürich)
Reverse Iterative Volume Sampling for Linear Regression abstract
Abstract:
Joint work with Michal Derezinski.
Consider the following basic one dimensional linear regression problem. You are given a set of n points and are to predict the hidden response value for each point. The goal is to achieve total square loss close to the optimallinear predictor without seeing most of the hidden responses. We show that if you sample JUST ONE point from the set and request its response value, then the linear predictor fitting that point response pair has exactly twice the optimum total square loss. In d dimensions, sampling a subset of just d points and fitting their responses achieves total loss d+1 times the optimum. The key trick is to use a joint sampling technique called volume sampling for sampling a diverse subset of points. We show that the least squares solution obtained for the volume sampled subproblem is an unbiased estimator of the optimal solution based on all n responses. This unbiasedness is a desirable property that is not shared by other common subset selection techniques.
Motivated by these basic properties, we develop a theoretical framework for studying volume sampling, resulting in a number of new matrix expectation equalities and statistical guarantees which are of importance not only to least squares regression but also to numerical linear algebra in general. Our methods also lead to a regularized variant of volume sampling, and we propose the first efficient algorithm for volume sampling which makes this technique a practical tool in the machine learning toolbox. Finally, we provide experimental evidence which confirms our theoretical findings.
12:30 • ETH Zentrum, Rämistrasse 101, Zürich, Building HG, Room D 5.2
Prof. Dr. Ben Bakker (University of Georgia)
o-minimal GAGA abstract
Abstract:
For a complex projective variety, Serre\'s classical GAGA theorem asserts that the analytification functor from algebraic coherent sheaves to analytic coherent sheaves is an equivalence of categories. For non-proper varieties, however, this theorem easily fails. In joint work with Y. Brunebarbe and J. Tsimerman, we show that a GAGA theorem holds even in the non-proper case if one restricts to analytic structures that are "tame" in a sense made precise by the notion of o-minimality. We will also discuss some related algebraization theorems.
13:30 • ETH Zentrum, Rämistrasse 101, Zürich, Building HG, Room G 43
Mirko Mauri (Imperial College London)
The essential skeleton of pairs and the geometric P=W conjecture abstract
Abstract:
The geometric P=W conjecture is a conjectural description of the asymptotic behavior of a celebrated correspondence in non-abelian Hodge theory. In particular, it is expected that the dual boundary complex of the compactification of character varieties is a sphere. In a joint work with Enrica Mazzon and Matthew Stevenson, we manage to compute the first non-trivial examples of dual complexes in the compact case. This requires to develop a new theory of essential skeletons over a trivially-valued field. As a byproduct, inspired by these constructions, we show that certain character varieties appear in degenerations of compact hyper-Kähler manifolds. In this talk we will explain how these new non-archimedean techniques can shed new light into classical algebraic geometry problems.
14:00 • Université de Genève, Villa Battelle, 7 route de Drize, 1227 Carouge
Tobias Weth (Universität Frankfurt)
The Dirichlet Problem for the Logarithmic Laplacian abstract
Abstract:
I will report on some recent results - obtained in joint work with Huyuan Chen - on Dirichlet problems for the Logarithmic Laplacian Operator, which arises as formal derivative of fractional Laplacians at order s= 0. I will discuss the functional analytic framework for these problems and show how it allows to characterize the asymptotics of principal Dirichlet eigenvalues and eigenfunctions of fractional Laplacians as the order tends to zero. Furthermore, I will discuss necessary and sufficient conditions on domains giving rise to weak and strong maximum principles for the logarithmic Laplacian. If time permits, I will also discuss regularity estimates for solutions to corresponding Poisson problems.
14:15 • Universität Basel, Fachbereich Mathematik, Spiegelgasse 1, Room 00.003
Prof. Dr. Johan Rosenkilde (Technical University of Denmark)
Twisted Reed-Solomon Codes abstract
Abstract:
Twisted Reed-Solomon codes are a new family of codes coming from the evaluation of "twisted polynomials": degree (k-1)-polynomials added with a few high-degree terms whose coefficients depend on the first k coefficients. This usually won\'t lead to an MDS code, so the principal game is to devise ways of choosing these parameters so the resulting codes are MDS. The construction was originally inspired by Sheekey\'s twisted Gabidulin codes, but is much more general and uses different techniques for proving the MDS-property. We present some infinite families of twisted RS codes which are MDS, and we discuss some properties of the family of twisted RS codes. We then go on to discuss some preliminary results on the decoding of twisted RS codes.
15:00 • UZH Irchel, Winterthurerstrasse 190, Zürich, Building Y27, Room H 28
Adrien Boyer (Université Paris Diderot, Paris 7)
Spherical functions on hyperbolic groups and property RD (Rapid Decay) abstract
Abstract:
We investigate properties of some spherical fonctions defined on hyperbolic groups using boundary representations on the Gromov boundary endowed with the Patterson-Sullivan measure class. We prove sharp decay estimates for spherical functions as well as spectral inequalities associated with boundary representations. This point of view on the boundary allows us to view the so-called property RD (Rapid Decay, also called Haagerup\'s inequality) as a particular case of a more general behavior of spherical functions on hyperbolic groups. If time permits, I will try to explain the idea of a constructive proof using a boundary unitary representation of a result due to de la Harpe and Jolissaint asserting that hyperbolic groups satisfy property RD.
15:45 • ETH Zentrum, Rämistrasse 101, Zürich, Building HG, Room G 43
Prof. Dr. Francisco Javier Lobillo Borrero (Universidad de Granada)
Projective codes and free distance abstract
Abstract:
\\(\\sigma\\)-Quasicyclic and projective convolutional codes are introduced. They are kernels of idempotent endomorphisms on free modules over \\(R = A[z;\\sigma]\\), where \\(A\\) is an algebra over a given finite field. The matrix structure of the endomorphism is used to defined cyclic column distances. They are used to compute the free distance of a projective convolutional code.
16:00 • UZH Irchel, Winterthurerstrasse 190, Zürich, Building Y27, Room H 28
Prof. Dr. Robert Scheichl (IWR, Uni Heidelberg)
Multilevel Uncertainty Quantification with Sample-Adaptive Model Hierarchies abstract
Abstract:
Sample-based multilevel uncertainty quantification tools, such as multilevel Monte Carlo, multilevel quasi-Monte Carlo or multilevel stochastic collocation, have recently gained huge popularity due to their potential to efficiently compute robust estimates of quantities of interest (QoI) derived from PDE models that are subject to uncertainties in the input data (coefficients, boundary conditions, geometry, etc). Especially for problems with low regularity, they are asymptotically optimal in that they can provide statistics about such QoIs at (asymptotically) the same cost as it takes to compute one sample to the target accuracy. However, when the data uncertainty is localised at random locations, such as for manufacturing defects in composite materials, the cost per sample can be reduced significantly by adapting the spatial discretisation individually for each sample. Moreover, the adaptive process typically produces coarser approximations that can be used directly for the multilevel uncertainty quantification. In this talk, I will present two novel developments that aim to exploit these ideas. In the first part I will present Continuous Level Monte Carlo (CLMC), a generalisation of multilevel Monte Carlo (MLMC) to a continuous framework where the level parameter is a continuous variable. This provides a natural framework to use sample-wise adaptive refinement strategy, with a goal-oriented error estimator as our new level parameter. We introduce a practical CLMC estimator (and algorithm) and prove a complexity theorem showing the same rate of complexity as for MLMC. Also, we show that it is possible to make the CLMC estimator unbiased with respect to the true quantity of interest. Finally, we provide two numerical experiments which test the CLMC framework alongside a sample-wise adaptive refinement strategy, showing clear gains over a standard MLMC approach with uniform grid hierarchies. In the second part, I will show how to extend the sample-adaptive strategy to multilevel stochastic collocation (MLSC) methods providing a complexity estimate and numerical experiments for a MLSC method that is fully adaptive in the dimension, in the polynomial degrees and in the spatial discretisation.
This is joint work with Gianluca Detommaso (Bath), Tim Dodwell (Exeter) and Jens Lang (Darmstadt).
16:15 • UZH Irchel, Winterthurerstrasse 190, Zürich, Building Y27, Room H 25
Denis Villemonais (Université de Nancy, France)
General criteria for the study of quasi-stationarity abstract
Abstract:
In this work in collaboration with Nicolas Champagnat (IECL, Nancy France), we provide a new set of criteria for proving the existence of quasi-stationary distributions for absorbed Markov processes typically evolving in non-compact spaces. The talk will go through the definition and basic properties of quasi-stationary distributions, the presentation of the above mention criteria and applications to perturbed dynamical systems, diffusion processes, measured valued Pólya processes (a collaboration with Cécile Mailler, Univeristy of Bath UK) and branching processes.
17:15 • UZH Irchel, Winterthurerstrasse 190, Zürich, Building Y27, Room H 12