Random Numbers, Random Variables and Compiling Graphical Models¶
Objective¶
It might be nice to use Theano as a language and compiler for questions about graphical models.
In this way, we could express something like Logistic Regression like this:
from theano import random_variable as RV
X, Y, s_idx = RV.empirical(my_dataset)
# model parameters
v = shared(numpy.zeros(()))
b = shared(numpy.zeros(()))
Y_hat = RV.multinomial(n=1, p=softmax(dot(X,v)+b))
cost = sum(-log(Y_hat.density(Y)))
train_fn = function([s_idx], cost, updates=[[v,b], grad(cost, [v,b]]))
RandomVariable(Variable)
def sample(self, n):
"""[Symbolically] draw a sample of size n"""
def density(self, pt, givens=None):
"""Conditional Density/Probability of P(self=pt)
Implicitly conditioned on knowing the values of all variables
on which this one depends. Optionally override ancestor variables
using givens.
"""
def mode(self):
"""Return expression of the most likely value of this distribution"""
We would really like to integrate out certain variables sometimes...
An RBM could be expressed like this:
w = shared(initial_weights)
v = shared(initial_visible_biases)
u = shared(initial_hidden_biases)
visible = RV.binomial(n=1, p=None) # p filled in by EnergyModel
hidden = RV.binomial(n=1, p=None) # p filled in by EnergyModel
energy = dot(visible,v) + dot(hidden, u) + dot(dot(visible, w), hidden)
RBM = EnergyModel(energy, variables={'visible':visible, 'hidden':hidden], params=[w,v,u])
RBM.energy(v,h) # an expression for the energy at point (v,h)
RBM.visible.energy(h) # an expression for the free energy
RBM.hidden.energy(h) # an expression for the free energy
v_given_h = RBM.visible.conditional(h) # a random variable
Rather than program all the training algorithms into an RBM module, the idea would be to express the relationship between RBM variables so that we could automatically recognize how to do Gibbs sampling, gradient descent on Free Energy, etc.