Aapo Hyvärinen: Brain Imaging at rest: the ultimate neuroscience data set?

Aapo Hyvärinen from Helsinki University gave his plenary talk about the study of brains at rest. There are a lot of studies in neuroscience, but why is it so interesting to study brains at rest? Subject is said to be at rest, when he is not performing a given task or he is not stimulated in any other way by the researcher. First of all, there are not many analyses done yet on this topic, though the measurements are easy to repeat and there are no time-limits. In addition, it is more objective because it is free from the researchers experimental design. A new view point is to try to learn more of the brains internal dynamics with this study. Research shows that some parts of the brain, called the default network, are even more active during rest than during stimulation. Maybe this way it is possible find out the ultimate neuroscience data set.

The measurement for studying brains are traditionally done with Electroencephalography (EEG), Magnetoencephalography (MEG) or Functional magnetic resonance imaging (fMRI). Supervised methods can not be used to analyze this data and the most popular unsupervised method is independent component analyses (ICA). It is used to find components by maximizing sparsity of a given variable. Hyvärinen also presented a spatial version of ICA, that is often used with fMRI, and how it could be used in MEG. ICA has been used to find resting state networks in fMRI with good results. The results were very similar to ones acquired from research with very complex stimulation: movies.

Hyvärinen highlights the importance of testing significance of the results. ICA itself does not provide information of its result reliability, but there are ways to test it statistically: do a separate ICA on several subjects and pick the significant components which appear in two or more subjects. It is possible that all significant components won't appear in all subjects. Part of the analysis is seeking connections between the measured variables. Hyvärinen explained different approaches used, for example, structural equation models and when those can be estimated.

Exploratory data analysis with ICA could bring us a better understanding of functioning of brains. Hyvärinen proposes that these methods could be more used in studies, where complicated stimulations are used. He admits that speaking of the ultimate data set here is an overstatement, and until we can properly do two person neuroscience, we can not fully understand the human brains, like Riitta Hari said in her plenary talk.

Enrique Romero: Using the Leader Algorithm with Support Vector Machines for Large Data Sets

One of the problems of Support Vector Machines (SVM) is that applying them to large datasets is computationally expensive. The computational cost often increases proportional to N^3, where N is the number of training samples.

Many different approaches to the problem have been proposed. E.g. chunking and decomposition methods optimize the SVM with respect to subsets of the data to lower the computational cost.

Romero presents an approach that aims to reduce the computational cost by reducing the number of training samples. The data set is first clustered using the Leader algorithm, and then only the samples chosen as the cluster identities by the Leader algorithm are used for training the SVM.

The Leader algorithm uses a distance measure D and and a predefined threshold T to partition the data. Neighborhoods that are withing distance T with respect to the distance measure D are clustered together and the cluster is represented using one of its data points, which is then referred to as the leader. The Leader algorithm is very fast: the algorithm makes a single pass through the dataset. All areas of the input space are presented in the clustering solution.

Reducing the size of the training set naturally decreases predictive performance but the computational cost decreases much more rapidly. As a future step, Romero proposes developing the Leader algorithm to preserve more data points close to the decision boundaries of the SVM.

Joshua Tenenbaum: How to grow a mind: Statistics, structure and abstraction

Joshua Tenenbaum is the Associate Professor of Computational Cognitive Science at MIT. On Friday, the last day of ICANN 2011, he gave an inspiring plenary presentation about reverse-engineering learning and cognitive development.

He stated that the most perplexing quality of the brain from machine learning perspective is its ability to grasp abstract concepts and infer causal relations with such sparse data, i.e "how does the mind get so much from so little?". He gave an entertaining example of this by showing a grid full of pictures of computer generated unidentifiable objects and naming three of them as "tufas". He then pointed at other objects in the grid and asked the audience whether it was or wasn't a tufa. There was a strong consensus and the answers were quite confidently "yes" or "no".

To explore how this kind of inference could be possible, Tenenbaum focused on what he called abstract knowledge. His talk was then divided into three parts, answering three different questions about abstract knowledge.

How does abstract knowledge guide learning and inference from sparse data?

According to Tenenbaum, the mind learns and reasons according to Bayesian principles. Simply put, there exists some sort of generative model of data and hypotheses and the probability of a certain hypothesis given data is given by the Bayes' rule. The abstract background knowledge affects the model through the available hypotheses and in the prior probabilities given to these hypotheses. The likelihood gives the probability of the data given a hypothesis.

What forms does abstract knowledge take?

It doesn't seem feasible to assume that every logically possible hypothesis is somehow presented along with its prior and likelihood. The hypotheses need to be presented in a more structured way. As Tenenbaum puts it: "some more sophisticated forms of knowledge representation must underlie the probabilistic generative models needed for Bayesian cognition".

Causes and effects can be modeled in a general way with directed graphs. As an example, in a symptom-disease model we would have symptoms and diseases as nodes and edges running from the diseases to the symptoms. The role of background knowledge here would be to know that there are two kinds of nodes and that the edges always run from diseases to symptoms, in effect limiting amount of hypotheses to be considered.

On the other hand it seems that tree structured representations would be most effective for learning words and concepts from examples.

How is abstract knowledge acquired?

So it seems that abstract background knowledge is required to make learning possible. But how then is this background knowledge learned? How does one know when to use a tree structured presentation and when is some other form more suitable?

Tenenbaum presented the answer in hierarchical Bayesian models or HBMs. They enable hypotheses spaces of hypothesis spaces and priors on priors. More specifically, Tenenbaum proceeded to show how HBMs can be used to infer the form (e.g. tree, ring, chain) and the structure simultaneously. An impressive example was sorting synthesized faces varying in race and masculinity into a correct matrix structure, where race varied along the other axis and masculinity along the other.

Conclusion

Clearly one of the goals of the talk was to establish that abstract background knowledge is essential in human learning. Its role is to constrain the logically valid hypotheses to make learning possible. Human learning was then formulated as Bayesian inference over richly structured hierarchical generative models.

Ramya Rasipuram and Mathew Magimai Doss: Improving Articulatory Feature and Phoneme Recognition using Multitask Learning

Articulatory features define properties of speech production, i.e. they describe the basic sounds we make. Phonemes on the other hand are the smallest units of sound used to form meaningful speech. In Finnish basically all the phonemes correspond to a letter, whereas in English they do not. However, phonemes are used to model pronounciation, and they are therefore cross- and multilingual.

The authors did experiments on their model using the TIMIT corpus, containing speech from American English speakers of different sexes and dialects. The corpus also contains the correct phonemes used in the speech. Following methods for phoneme recognition were applied:

1. Independent MLP (multilayer perceptron)
2. Multitask MLP
3. Phoneme MLP

Independent MLP is a standard method, whereas (2) and (3) are novel methods presented in their paper. In each method, articulatory features were learned from the audio, and an MLP network was trained to predict the phonemes. In independent MLP the classifiers are independent. However, since the features actually are interrelated, multitask learning was considered to be needed. The prediction accuracies (speech to phoneme) for independent, multitask and phenome MLP were 67.4%, 68.9% and 70.2%, respectively.

Additionally, a hierarchical version was presented for each method. They performed better than the original ones, maintaining the order of performance.

Rasipuram presented their work to be continued with:

• Automatic speech recognition studies
• Different importance weights for features
• Adding gender and rate of speech as features

The talk gained some critique, as one researcher in the audience stated that performance better than this had been achieved already years ago. This wasn't really addressed by the author.