Table Of Contents

Previous topic

Automatic updates

Next topic

Random Numbers, Random Variables and Compiling Graphical Models

This Page

Optimization Patterns

Motivation

Theano optimizations are organized at high level, but canonicalization and specialization (C&S) are a mess. It is difficult to know how a graph will be optimized, or to know in which order optimizations will be performed. C&S is also slow because of the guess-and-check nature of node optimization within equilibrium optimizers (VERIFY THIS BY PROFILING). C&S functions are also very difficult and tedious to write because of symmetries in the graph, and because of the lack of standard Op names (e.g. T.add, T.and_, and T._shape). Gemm and the advanced_indexing -> xent optimization are particularly tricky examples.

Defining a sort of regexp-like approach for describing graph substitutions would ideally be less error-prone, less tedious, more efficient to evaluate, easier to document, and all-round better.

Proposal

In a nutshell: revisit the PatternSub and make it more powerful.

Olivier B. (original author or PatternSub) mentioned that one of the problems was the annoyance of working through DimShuffle

Olivier B. also suggests writing scalar-related patterns in terms of scalars, and then inferring Tensor-related patterns.