The article Why Machine Learning Doesn’t Work Well for Some Problems?(Shahab , 2017) describes the effect of Emergence as a barrier for predictive inference.

Emergence is the phenomenon of completely new behavior arising (emerging) from interactions of elementary entities, such as life emerging from biochemistry and collective intelligence emerging from social animals.

In general, effects of emergence cannot be inferred through a priori analysis of a system (or its elementary entities). While weak emergence can be understood still by observing or simulating the system, emergent qualities from strong emergence cannot be simulated with current systems.

Sheikh-Bahei suggests interpreting emergence (in a predictive context) as an additional dimension, called the E-Dimension, where moving along that dimension results in new qualities emerging. Crossing E-Dimensions during inferrence leads to reduced predictive power as emergent qualities cannot be necessarily described as a function of the observed features alone. The more E-Dimensions are crossed during inferrence, the lower the prediction success will be, regardless of the amount of feature noise. Current-generation algorithms do not handle this kind of problem well and further research is required in this area.

Hypothetical example of the E-Dimension concept: Emergence phenomena can be considered as a barrier for making predictive inferences. The further away the target is from features along this dimension, the less information the features provide about the target. The figure shows an example of predicting organism level properties (target) using molecular and physicochemical properties (feature space). (Shahab , 2017)

Effects of emergence on example machine learning problems (Shahab , 2017):

One of TensorFlow’s more awesome parts is definitely TensorBoard, i.e. the capability of collecting and visualizing data from the TensorFlow graph as the network is running while also being able to display and browse the graph itself. Coming from Caffe, where I eventually wrote my own tooling just to visualize the training loss from logs of the raw console output and hat to copy-paste the graph’s prototxt to some online service in order to visualize it, this is a massive step in the best possible direction. To get some of Caffe’s checkpointing features back, you can use TensorFlow’s Supervisor. This blog post is about using both TensorBoard and the Supervisor for fun and profit. TL;DR: Scroll to the end for an example of using grouped summaries with the Supervisor.

Apart from just storing scalar data for TensorBoard, the histogram feature turned out to be especially valuable to me for observing the performance of a probability inference step.

Here, the left half shows the distribution of ground truth probability values in the training and validation sets over time, whereas the right half shows the actual inferred probabilities over time. It’s not hard to see that the network is getting better, but there is more to it:

The histogram of the ground truth values (here on the left) allows you to verify that your training data is indeed correct. If the data is not balanced, you might learn a network that is biased towards one outcome.

If the network does indeed obtain some biased view of the data, you’ll cleary see patterns emerging in the inferred histogram that do not match the expected ground truth distribution. In this example, the right histograms approach the left histograms, so it appears to be working fine.

However, if you only measure network performance in accuracy, as ratio of correct guesses over all examples, you might be getting the wrong impression: If the input distribution is skewed towards 95% positive and 5% negative examples, a network guessing “positive” 100% of the time is producing only 5% error. If your total accuracy is an aggregate over multiple different values, you will definitely miss this, especially since randomized mini-batches only further obscure this issue.

Worse, if the learned coefficients run into saturation, learning will stop for them. Again, this might not be obvious if the total loss and accuracy is actually an aggregate of different values.

Influence of the learning rate

Let’s take the example of a variable learning rate. If at some point the training slows down, it’s not immediately clear if this is due to the fact that

a parameter space optimum has been found and training is done,

the algorithm found a plateau in parameter space and would continue to fall after a few more hundreds or thousands of iterations or

the training is actually diverging because the learning rate is not small enough in order to enter a local optimum in the first place.

Now optimizers like Adam are tailored to overcome the problems of fixed learning rates but they too can only go so far: If the learning rate is too big to begin with, it’s still too big after fine-tuning. Or worse, after a couple of iterations the adjusted weights could end up in saturation and no further change would be able to do anything to change this.

To rule out at least one part, you can make the learning rate a changeable parameter of the network, e.g. a function of the training iteration. I had some success in using Caffe’s “multi-step” approach of changing the learning rate at fixed iteration numbers — say, reducing it one decade at iteration 1000, 5000 and 16000 — where I determined these values over different training runs of the network.

So instead of baking the learning rate into the graph during construction, you would define a placeholder for it and feed the learning rate of the current epoch/iteration into the optimization operation each time you call it, like so:

with tf.Graph().as_default() as graph:
p_lr = tf.placeholder(tf.float32, (), name='learning_rate')
t_loss = tf.reduce_mean(...)
op_minimize = tf.train.AdamOptimizer(learning_rate=p_lr)\
.minimize(t_loss)
with tf.Session(graph=graph) as sess:
init = tf.group(tf.global_variables_initializer(),
tf.local_variables_initializer())
sess.run(init)
for _ in range(0, epochs):
learning_rate = 0.1
loss, _ = sess.run([t_loss, op_minimize],
feed_dict={p_lr: learning_rate)

Alternatively, you could make it a non-learnable Variable and explicitly assign it whenever it needs to be changed; let’s assume we don’t do that.
The first thing I usually do is then to also add a summary node to track the current learning rate (as well as the training loss):

with tf.Graph().as_default() as graph:
p_lr = tf.placeholder(tf.float32, (), name='learning_rate')
t_loss = tf.reduce_mean(...)
op_minimize = tf.train.AdamOptimizer(learning_rate=p_lr)\
.minimize(t_loss)
tf.summary.scalar('learning_rate', p_lr)
tf.summary.scalar('loss', t_loss)
# histograms work the same way
tf.summary.histogram('probability', t_some_batch)
s_merged = tf.summary.merge_all()
writer = tf.summary.FileWriter('log', graph=graph)
with tf.Session(graph=graph) as sess:
init = tf.group(tf.global_variables_initializer(),
tf.local_variables_initializer())
sess.run(init)
for _ in range(0, epochs):
learning_rate = 0.1
loss, summary, _ = sess.run([t_loss, s_merged, op_minimize],
feed_dict={p_lr: learning_rate)
writer.add_summary(summary)

Now, for each epoch, the values of the t_loss and p_lr tensors are stored in a protocol buffer file in the log subdirectory. You can then start TensorBoard with the --logdir parameter pointing to it and get a nice visualization of the training progress.

And one example where doing this massively helped me tracking down errors is exactly the network I took the introduction histogram picture from; here, I set the learning rate to 0.1 for about a two hundred iterations before dropping it to 0.01. It turned out that having the learning rate this high for my particular network did result in saturation and learning effectively stopped. The histogram helped noticing the issue and the scalar graph helped determining the “correct” learning rates.

Training and validation set summaries

Suppose now you want to have different summaries that may or may not appear on different instances of the graph. The learning rate, for example, has no influence on the outcome of the validation batch, so including it in validation runs is only eating up time, memory and storage. However, the tf.summary.merge_all() operation doesn’t care where the summaries live per se — and since some summaries depend on nodes from the training graph (e.g. the learning rate placeholder), you suddenly create a dependency on nodes you didn’t want to trigger — with effects of very varying levels of fun.

It turns out that summarries can be bundled into collections — e.g. “train” and “test” — by specifying their membership upon construction, so that you can later obtain only those summaries that belong to the specified collections:

with tf.Graph().as_default() as graph:
p_lr = tf.placeholder(tf.float32, (), name='learning_rate')
t_loss = tf.reduce_mean(...)
op_minimize = tf.train.AdamOptimizer(learning_rate=p_lr)\
.minimize(t_loss)
tf.summary.scalar('learning_rate', p_lr, collections=['train'])
tf.summary.scalar('loss', t_loss, collections=['train', 'test'])
# merge summaries per collection
s_training = tf.summary.merge_all('train')
s_test = tf.summary.merge_all('test')
writer = tf.summary.FileWriter('log', graph=graph)
with tf.Session(graph=graph) as sess:
init = tf.group(tf.global_variables_initializer(),
tf.local_variables_initializer())
sess.run(init)
for _ in range(0, epochs):
# during training
learning_rate = 0.1
loss, summary, _ = sess.run([t_loss, s_training, op_minimize],
feed_dict={p_lr: learning_rate)
writer.add_summary(summary)
# during validation
loss, summary = sess.run([t_loss, s_test])
writer.add_summary(summary)

In combinaion with liberal uses of tf.name_scope(), it could then look like on the following image. The graphs shows three different training runs where we now got the ability to reason about the choice(s) of the learning rate.

This works, but we can do better.

Using the Supervisor

One currently (documentation wise) very underrepresented yet powerful feature of TensorFlow’s Python API is the Supervisor, a manager that basically takes care of writing summaries, taking snapshots, running queues (should you use them, which you probably do), initializing variables and also gracefully stopping training.

In order to use the Supervisor you basically swap out your own session with a managed one, skip variable initialization and tell it when you want which of your custom summaries to be stored. While not being required, but apparently being a good practice is the addition of a global_step variable to the graph; should the Supervisor find such a variable, it will automatically use it for internal coordination. If you bind the variable to the optimizer it will also be automatically incremented for each optimization step, freeing you from having to keep track of the iteration yourself. Here’s an example of how to use it:

with tf.Graph().as_default() as graph:
p_lr = tf.placeholder(tf.float32, (), name='learning_rate')
t_loss = tf.reduce_mean(...)
# adding the global_step and telling the optimizer about it
global_step = tf.Variable(0, name='global_step', trainable=False)
op_minimize = tf.train.AdamOptimizer(learning_rate=p_lr)\
.minimize(t_loss, global_step=global_step)
tf.summary.scalar('learning_rate', p_lr, collections=['train'])
tf.summary.scalar('loss', t_loss, collections=['train', 'test'])
s_training = tf.summary.merge_all('train')
s_test = tf.summary.merge_all('test')
# create the supervisor and obtain a managed session;
# variable initialization will now be done automatically.
sv = tf.train.Supervisor(logdir='log', graph=graph)
with sv.managed_session() as sess:
# run until training should stop
while not sv.should_stop():
learning_rate = 0.1
loss, s, i, _ = sess.run([t_loss, s_training,
global_step, op_minimize],
feed_dict={p_lr: learning_rate)
# hand over your own summaries to the Supervisor
sv.summary_computed(sess, s, global_step=i)
loss, s = sess.run([t_loss, s_test])
sv.summary_computed(sess, s, global_step=i)
# ... at some point, request a stop
sv.request_stop()

The Supervisor will also add additional summaries to your graph for free, e.g. an insight over the number of training steps per second. This could allow you to fine-tune minibatch sizes, for example, because they currently tend to have a big impact on the host to device transmission on the data.
Different from Caffe’s behavior, the Supervisor will by default keep only the last five snapshots of the learned weights; unless you fear of missing the validation loss optimum, leaving the training running for days is now not an issue anymore — diskwise, at least.

Januar 21st, 2017 GMT +2 von
Markus
2017-01-21T18:45:45+02:002017-02-2T03:34:54+02:00
· 2 Kommentare

Let’s assume you already have an image in numpy’s ndarray format, e.g. because you loaded it with OpenCV’s imread() function, and you want to convert it to TensorFlow’s Tensor format and later back to ndarray.

That’s essentially three calls to TensorFlow:

import cv2
import tensorflow as tf
import numpy as np
# normalize the pixel values to 0..1 range and convert them
# to a single-precision tensor
image_in = cv2.imread('image.png') / 255.
t = tf.convert_to_tensor(image_in, dtype=tf.float32)
assert isinstance(t, tf.Tensor)
# in order to convert the tensor back to an array, we need
# to evaluate it; for this, we need a session
with tf.Session() as sess:
image_out = sess.run(fetches=t)
assert isinstance(image_out, np.ndarray)
# for imshow to work, the image needs to be in 0..1 range
# whenever it is a float; that's why we normalized it.
cv2.imshow('Image', image_out)
cv2.readKey(0)

Note that instead of using sess.run(t) we could also have used

with tf.Session() as sess:
image_out = t.eval(sess)

which essentially performs the same action. A benefit of using sess.run() directly is that we can fetch more than one tensor in the same pass through the (sub-)graph (say, tuple = sess.run(fetches=[t1, t2, t3])), whereas calling tensor.eval() always results in one separate pass per call.

Dezember 12th, 2016 GMT +2 von
Markus
2016-12-12T15:59:54+02:002016-12-12T15:59:54+02:00
· 0 Kommentare

Note that now that BVLC’s caffe repository directly supports compilation on Windows, this guide has become obsolete.

When attempting to check out Caffe after some wasted hours of getting TensorFlow to run on Windows (which sort-of works using Bash for Windows), I gave Caffe a try. However, the last time I tried to get a huge system of nested dependencies using CMake to work on Windows, things only got worse with every additional project. For Caffe, you’d need Boost, OpenCV, HDF and protobuf as well as some Google logging and command-line argument things amongst other stuff — and boy, make sure you got your linking, threading and runtime right, and don’t even think about mixing different C++ or Boost flavors.

So there’s this Windows fork of Caffe managed by Microsoft which apparently only requires you to configure the CommonSettings.props file to your liking and then compile. The nice part is, all references are pulled using NuGet. The bad part is, the packages are all MSVC 1800, i.e. Visual Studio 2013. You’ll be able to find newer variants of Boost, but that’s about it.

If you’re really trying to use Visual Studio 2015, fix the nuget.config to have the repository path fixed like described in this StackOverflow answer:

Also note that the path is most likely outside your build tree … but that’s how it is provided by the maintainer.
I wasted another day trying to get Caffe to work in Visual Studio 2015 by selecting the 2013 toolkit but, in the end, it came down to the following error:

NuGet error: unknown command 'overlay'

This overlay stuff apparently is some NuGet 2 magic that didn’t survive the dark ages, so it won’t work with NuGet 3, which I have in my path and which is also used internally by VS 2015.
There are a couple of questions regarding that error, like this one on the OpenCV Answers site, issues on GitHub (with half-broken answers since the CoApp tooling doesn’t really work exist anymore), etc., but there’s no obvious solution — at least if you’re using a modern Windows development environment — that is, one that’s using NuGet 3 instead of NuGet 2.6.

You can fix most of the problems by just upgrading OpenCV 2 to OpenCV 3. For that, kick out the old NuGet packages and reference the opencvdefault one as shown in this video. But even if you do that, it’ll fail on the glog 0.3.3 NuGet package which still uses overlays.

As it turns out though, the NuGet package for glog 0.3.3 already contains a NuGet binary at glog.0.3.3.0/build/native/private/nuget.exe, but since it’s not in the path, it won’t use it. Copy that to the package’s root (where NuGet-Overlay.cmd can be found and you’re set.

For the rest, make sure you’re using Python 2.7 though, as the Boost libraries are requiring that; also install NumPy.
Using e.g. Anaconda Python, you’d want

conda create -n caffe python=2.7 numpy

Which will leave you with the task to find python27_d.lib … or just build Release.

September 1st, 2016 GMT +2 von
Markus
2016-09-1T12:46:36+02:002016-12-12T16:03:10+02:00
· 0 Kommentare

While reading up on line search algorithms in nonlinear optimization for neural network training, I came across this problem: Given a function \(f(x)\) , find a quadratic interpolant \(q(x) = ax^2 + bx + c\) that fulfills the conditions \(f(x_0) = q(x_0)\) , \(f(x_1) = q(x_1)\) and \(f'(x_0) = q'(x_0)\) . Basically this:

So I took out my scribbling pad, wrote down some equations and then, after two pages of nonsense, decided it really wasn’t worth the hassle. It turns out that the simple system

We also would need to check the interpolant’s second derivative \(q''(x_{min})\) to ensure the approximated minimizer is indeed a minimum of \(q(x)\) by requiring \(q''(x_{min}) > 0\) , with the second derivative given as:

The premise of the line search in minimization problems usually is that the search direction is already a direction of descent. By having \(0 > f'(x_0)\) and \(f'(x_1) > 0\) (as would typically be the case when bracketing the local minimizer of \(f(x)\) ), the interpolant should always be (strictly) convex. If these conditions do not hold, there might be no solution at all: one obviously won’t be able to find a quadratic interpolant given the initial conditions for a function that is linear to machine precision. In that case, watch out for divisions by zero.

Last but not least, if the objective is to minimize \(\varphi(\alpha) = f(\vec{x}_k + \alpha \vec{d}_k)\) using \(q(\alpha)\) , where \(\vec{d}_k\) is the search direction and \(\vec{x}_k\) the current starting point, such that

If \(q(\alpha)\) is required to be strongly convex, then we’ll observe that

\begin{align}
q''(\alpha) &= 2a \overset{!}{\succeq} m
\end{align}

for an \(m > 0\) , giving us that \(a\) must be greater than zero (or \(\epsilon\) , for that matter), which is a trivial check. The following picture visualizes that this is indeed the case:

Convexity of a parabola for different highest-order coefficients a with positive b (top), zero b (middle) and negative b (bottom). Lowest-order coefficient c is left out for brevity.

Juli 2nd, 2015 GMT +2 von
Markus
2015-07-2T04:54:49+02:002018-03-4T14:45:44+02:00
· 0 Kommentare