Basically, this file contains stuff that should be documented, but is not.
Feel free to contribute things that you want documented, as well as to add or correct documentation.
How do you define the grad function?¶
Let’s talk about defining the grad() function in an Op, using an illustrative example.
In Poisson regression (Ranzato and Szummer, 2008), the target t is integer data, which we predict using a continuous output o. In the negative log likelihood of the Poisson regressor, there is a term:
Let’s say we write a logfactorial Op. We then compute the gradient
You should define gradient, even if it is undefined. [give log factorial example]
If an Op does not define grad
, but this Op does not appear in the path when
you compute the gradient, then there is no problem.
If an Op does not define grad
, and this Op does appear in the path when
you compute the gradient, WRITEME.
Gradients for a particular variable can be one of four kinds: 1) forgot to implement it
You will get an exception of the following form.
theano.gof.utils.MethodNotDefined: (‘grad’, <class ‘pylearn.algorithms.sandbox.cost.LogFactorial’>, ‘LogFactorial’)
- a symbolic variable
- None / zero
- undefined mathematically
currently, there is no way for a grad() method to distinguish between cases 3 and 4 but the distinction is important because graphs with type-3 gradients are ok to run, whereas graphs with type-4 gradients are not. so I suggested that Joseph return a type-4 gradient by defining an Op with no perform method. the idea would be that this would suit the graph-construction phase, but would prevent linking. how does that sound to you?
This documentation is useful when we show users how to write Ops.
What is staticmethod, st_impl?¶
st_impl
is an optional method in an Op.
@staticmethod
is a Python decorator for a class method that does not
implicitly take the class instance as a first argument. Hence, st_impl
can be used for Op implementations when no information from the Op
instance is needed. This can be useful for testing an implementation.
See the XlogX
class below for an example.
This documentation is useful when we show users how to write Ops. Olivier says this behavior should be discouraged but I feel that st_impl should be encouraged where possible.
how do we write scalar ops and upgrade them to tensor ops?¶
Olivier says that XlogX
gives a good example. In fact, I would
like to beef xlogx up into our running example for demonstrating how to
write an Op:
class XlogX(scalar.UnaryScalarOp):
"""
Compute X * log(X), with special case 0 log(0) = 0.
"""
@staticmethod
def st_impl(x):
if x == 0.0:
return 0.0
return x * numpy.log(x)
def impl(self, x):
return XlogX.st_impl(x)
def grad(self, inp, grads):
x, = inp
gz, = grads
return [gz * (1 + scalar.log(x))]
def c_code(self, node, name, inp, out, sub):
x, = inp
z, = out
if node.inputs[0].type in [scalar.float32, scalar.float64]:
return """%(z)s =
%(x)s == 0.0
? 0.0
: %(x)s * log(%(x)s);""" % locals()
raise NotImplementedError('only floatingpoint is implemented')
scalar_xlogx = XlogX(scalar.upgrade_to_float, name='scalar_xlogx')
xlogx = tensor.Elemwise(scalar_xlogx, name='xlogx')
It is also necessary to talk about UnaryScalarOp vs. BinaryOp.
UnaryScalarOp is the same as scalar.ScalarOp with member variable nin=1. give an example of this
Documentation on how to write tests¶
Guillaume can you make sure to hit these points:
What are canonical examples of tests?
- What are the different test patterns?
nnet.py:
- What is going on with test1, test2, test3, test4?
What is the right eq function to use?
- There are a lot of tests that define their own epsilon, but this should be standardized. e.g. in test_elemwise.py
self.assertTrue((numpy.abs(f(xv) - zv) < 1e-10).all())
If the expected variable of a test is that an Exception is thrown, how do we correctly detect and handle that?
nosetests has
assertRaises
- Convention is that all test files must start with
test_
, not
_test_
, so rename all that use the old convention?
How to use the PrintOp¶
** This is also useful in the How to write an Op tutorial. **
Mammouth¶
This is internal documentation. Guillaume can you make sure to hit these points:
export THEANO_BLAS_LDFLAGS=’-lmkl -liomp5 -fopenmp’
Do we want the following:
export OMP_NUM_THREADS=2
Type checking¶
- Are there functions for doing type checking?
like dtype of this matrix is an int-type (not just int32 or int64) “if isinstance(item, int):” is the preferred way to do it in python now, so mimic this If the type is wrong, what exception should be raised?
More simple numpy stuff¶
- If we have a matrix with only one row, how do we convert it to a vector?
x.reshape(x.size)
You can also useresize
but there is not reason to ‘’resize’‘
- How do you convert the type of a numpy array?
theano._asarray(x, dtype = 'int32')
Note that usingnumpy.asarray
is potentially dangerous, due to a problem in numpy where the type may not be properly set (see numpy’s Track ticket #870).
How to reuse (overwrite) a storage tensor¶
theano.compile.io.Out(gw1, borrow = True)
for that value in
compile.function