Several important points made by the paper. The first is that:

• The asymptotic error made by logistic regression is no more than that made by naive Bayes (Proposition 1 in the paper).

This conclusion provides a basis for what seems to be the widely held belief that discriminative classifiers are better than generative ones. The main conclusions of the paper is about the sample complexity of both classifiers. Sample complexity is the number of examples needed to approach the asymptotic error. For logistic regression, such sample complexity is:
\[
m = \Omega(n)
\]which means that the sample complexity is linear in \( n \) (Proposition 2 in the paper). For naive Bayes, we have:
\[
m = O(\log n)
\]which means that the sample complexity is logarithmic in \( n \) (Lemma 3 and Corollary 6 in the paper). All these conclusions imply that even though naive Bayes converges to a higher asymptotic error compared to logistic regression, it may also approach it significantly faster — after \( O(\log n) \), rather than \( O(n) \), training examples.

These ideas are discussed in [1]. Are these theoretical analysis always hold in practice? Not very reliable as far as [2] suggested. Note that these conclusions coming after strong assumptions. Nevertheless, they provide some general idea between logistic regression and naive Bayes.

References:

[1] On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes, Andrew Y. Ng and Michael Jordan. In NIPS 14, 2002. [ps, pdf]
[2] Comment on “On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes”. Jing-Hao Xue and D. Michael Titterington. 2008. Neural Processing Letters 28, 3 (December 2008), 169-187. [pdf]

It is still a little bit difficult to follow up the ideas introduced in the slides. However, it has been demonstrated in the tutorial that many aspects of IR are developed or re-phrased under QT framework, such as relevance theory, mixture of models, multiple document summarization and relevance feedback.

The current biggest initiative of QM is Quantum Interaction conferences.

Firstly, this paper tries to introduce a new way to do inference for latent Dirichlet allocation (LDA). Traditionally, variational inference and Gibbs sampling are widely used to perform Bayesian inference for LDA. For a simpler version of LDA, Probabilistic Latent Semantic Analysis (PLSA), a Maximum Likelihood Estimation (MLE) via EM algorithm is usually used where no prior distributions are used at all. Both Bayesian inference and MLE can be treated as two extremes to learning a LDA or PLSA as MLE may be prone to over-fitting but Bayesian inference is a little bit overly complicated. The parameterization of the model is slightly different from normal settings. Let \( \mathbf{x}_{i} \) be a vector of counts in \( V \) categories (terms) for document \( i \). Basically, each element in this vector is the number of times term \( v \) appearing in document \( i \). We also have the total term count \( m_{i} = \sum_{j}^{V} x_{i,j} \). Thus, the \( K \)-topic model has the following generative story:

\[
\begin{aligned}
\mathbf{x}_{i} \sim \mbox{Multinomial}(\boldsymbol{\theta}_{i,1} \boldsymbol{\phi}_{1} + \cdots + \boldsymbol{\theta}_{i,K} \boldsymbol{\phi}_{K}, m_{i})
\end{aligned}
\]

where \( \boldsymbol{\theta}_{i} \) is a per-document distribution over topics and \( \boldsymbol{\phi}_{k} \) is a distribution over words. Of course, this setting is indeed exactly same as previously published settings. However, we can easily view the term counts as a function of total counts explicitly. Again, Dirichlet distributions are put on \( \boldsymbol{\theta} \) and \( \boldsymbol{\phi} \). The paper discusses a joint posterior maximization by using EM algorithm. The author treats the term count vector \( \mathbf{x}_{i} \) as:

\[
\begin{aligned}
\mathbf{x}_{i} \sim \mbox{Multinomial}(\boldsymbol{\phi}_{1}, t_{i,1}) + \cdots + \mbox{Multinomial}(\boldsymbol{\phi}_{K}, t_{i,K})
\end{aligned}
\]

where \( \mathbf{t}_{i} \sim \mbox{Multinomial}(\boldsymbol{\theta}_{i}, m_{i}) \) and these \( \mathbf{t}_{i} \) are treated as missing-data for EM algorithm. Thus, missing-data is no longer single topic assignments for each term but aggregated topic counts for the whole documents. The detailed derivations of the corresponding EM algorithm is in the paper.

The second point of the paper is to propose some new method to perform model selection, which is always difficult for topic models in some sense. The whole idea is to find \( K \) that can maximize \( P(\mathbf{X} \, | \, K) \), which is usually approximated through MCMC. In this paper, the author details Laplace’s method to approximate this probability. Again, details are in the paper. The last point made by the paper is to use multinomial dispersion to evaluate whether the model fits the data, which is standard in statistics but never used in topic modeling literature.

The experiments are somewhat interesting. The author demonstrated that MAP is almost always comparable to variational EM (VEM) and sometimes slightly better than Gibbs sampling. Sometimes, MAP is even better than VEM.

Note, this is NOT the first paper to propose a MAP learning for topic models. Here’s one at least:

J. T. Chien, J. T. Chien, M. S. Wu, and M. S. Wu. Adaptive Bayesian Latent Semantic Analysis. Audio, Speech, and Language Processing, IEEE Transactions on [seealso Speech and Audio Processing, IEEE Transactions on], 16(1):198–207, 2008.

Slides for VC Dimension.

For some reason, this feature is not supported by the current version of Tweepy (as of April 4, 2012). So, we need to hack a little bit to get the result. Indeed, this issue was discussed here. The current solution is to add the following code into “parsers.py” of the source code of Tweepy:

1
2
3

class RawJsonParser(Parser):
    def parse(self, method, payload):
        return payload

Therefore, we can call the following code for invoking a new parser with JSON returned:

1
2

from tweepy.parsers import RawJsonParser
api = tweepy.API(auth_handler=auth, parser=RawJsonParser())

Prerequisite:

1. Download the latest version of LAPACK and extracted it into path [LAPACK].
2. Download the latest version of BLAS and extracted it into path [BLAS].

Note that in almost all tutorials on how to install Numpy and Scipy on Linux machines discuss how to install them with ATALS. This is possible only if you have the root privilege where you can turn off CPU threshoding . Since we do not have root privilege, we can only install them with LAPACK and BLAS.

Step 1: Install Numpy

1. Edit “site.cfg”
   a) Enable “[DEFAULT]” section and add

   src_dirs = [BLAS]:[LAPACK]

   b) Add

   [blas_opt]
   libraries = f77blas, cblas
    
   [lapack_opt]
   libraries = lapack, f77blas, cblas

2. Type the following command in the shell:

   python setup.py build --fcompiler=gnu95

   which will compile the package with “gfortran”.

3. Type the following command in the shell:

   python setup.py install

Step 2: Install Scipy

Once Numpy is installed.  Scipy can be easily built and installed through normal “python setup.py build” and “python setup.py install” process. Remember that these command should be accompanied with “–fcompiler=gnu95″.

Here is the code for creating tuples and doing the sort. It is pretty straightforward.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#include <iostream>
#include <string>
#include <vector>
#include <tuple>
#include <algorithm>
using namespace std;
 
typedef tuple<string,double,int> mytuple;
 
bool mycompare (const mytuple &lhs, const mytuple &rhs){
  return get<1>(lhs) < get<1>(rhs);
}
 
int main(void){
  vector<mytuple> data;
  data.push_back(make_tuple("abc",4.5,1));
  data.push_back(make_tuple("def",5.5,-1));
  data.push_back(make_tuple("wolf",-3.47,1));
  sort(data.begin(),data.end(),mycompare);
  for(vector<mytuple>::iterator iter = data.begin(); iter != data.end(); iter++){
    cout << get<0>(*iter) << "\t" << get<1>(*iter) << "\t" << get<2>(*iter) << endl;
  }
}

The code is successfully compiled by G++ 4.6.1 with the option “-std=gnu++0x”.

Logistic Function & Logistic Regression

The common definition of Logistic Function is as follows:
\[
P(x) = \frac{1}{1+\exp(-x)} \;\; \qquad (1)
\] where \(x \in \mathbb{R} \) is the variable of the function and \(P(x) \in [0,1]\). One important property of Equation (1) is that:
\[ \begin{eqnarray}
P(-x) &=& \frac{1}{1+\exp(x)} \nonumber \\
&=& \frac{1}{1+\frac{1}{\exp(-x)}} \nonumber \\
&=& \frac{\exp(-x)}{1+\exp(-x)} \nonumber \\
&=& 1 - \frac{1}{1+\exp(-x)} \nonumber \\
&=& 1 - P(x) \; \; \qquad (2)
\end{eqnarray} \]The form of Equation (2) is widely used as the form of Logistic Regression (e.g., [1,2,3]):
\[ \begin{eqnarray}
P(y = 1 \, | \, \boldsymbol{\beta}, \mathbf{x}) &=& \frac{\exp(\boldsymbol{\beta}^{T} \mathbf{x})}{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})} \nonumber \\
P(y = 0 \, | \, \boldsymbol{\beta}, \mathbf{x}) &=& \frac{1}{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})} \;\; \qquad (3)
\end{eqnarray} \] where \(\mathbf{x}\) is a feature vector and \(\boldsymbol{\beta}\) is a coefficient vector. By using Equation (2), we also have:
\[ \begin{equation}
P(y=1 \, | \, \boldsymbol{\beta}, \mathbf{x}) = 1 - P(y=0 \, | \, \boldsymbol{\beta}, \mathbf{x})
\end{equation} \] This formalism of Logistic Regression is used in [1,2] where labels \( y \in \{0,1\} \) and the functional form of the probability to generate different labels is different. Another formalism introduced in [3] unified the two forms into one single equation by integrating the label and the prediction together:
\[ \begin{equation}
P(g= \pm 1 \, | \, \boldsymbol{\beta}, \mathbf{x}) = \frac{1}{1 + \exp( - g\boldsymbol{\beta}^{T} \mathbf{x})} \;\; \qquad (4)
\end{equation} \]where \( g \in \{\pm 1\} \) is the label for data item \( x \). It is also easily to verify that \( P(g=1 \, | \, \boldsymbol{\beta}, \mathbf{x}) = 1 – P(g=-1 \, | \, \boldsymbol{\beta}, \mathbf{x}) \).

The Equivalence of Two Forms of Logistic Regression

At first glance, the form (3) and the form (4) looks very different. However, the equivalence between these two forms can be easily established. Starting from the form (3), we can have:
\[ \begin{eqnarray}
P(y = 1 \, | \, \boldsymbol{\beta}, \mathbf{x}) &=& \frac{\exp(\boldsymbol{\beta}^{T} \mathbf{x})}{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})} \nonumber \\
&=& \frac{1}{\frac{1}{\exp(\boldsymbol{\beta}^{T} \mathbf{x})} + 1} \nonumber \\
&=& \frac{1}{\exp(-\boldsymbol{\beta}^{T} \mathbf{x}) + 1} \nonumber \\
&=& P(g= 1 \, | \, \boldsymbol{\beta}, \mathbf{x})
\end{eqnarray} \]We can also establish the equivalence between \( P(y=0 \, | \, \boldsymbol{\beta}, \mathbf{x})\) and \(P(g=-1 \, | \, \boldsymbol{\beta}, \mathbf{x})\) easily by using property (2). Another way to establish the equivalence is from the classification rule. For the form (3), we have the following classification rule:
\[ \begin{eqnarray}
\frac{\frac{\exp(\boldsymbol{\beta}^{T} \mathbf{x})}{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})}}{\frac{1}{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})}} & > & 1 \;\; \rightarrow \;\; y = 1 \nonumber \\
\exp(\boldsymbol{\beta}^{T} \mathbf{x}) & > & 1 \nonumber \\
\boldsymbol{\beta}^{T} \mathbf{x} & > & 0
\end{eqnarray} \]An exactly same classification rule for the form (4) can also be obtained as:
\[ \begin{eqnarray}
\frac{\frac{1}{1 + \exp( - \boldsymbol{\beta}^{T} \mathbf{x})}}{\frac{1}{1 + \exp( \boldsymbol{\beta}^{T} \mathbf{x})}} & > & 1 \;\; \rightarrow \;\; g = 1 \nonumber \\
\frac{1 + \exp(\boldsymbol{\beta}^{T} \mathbf{x})}{1 + \exp( - \boldsymbol{\beta}^{T} \mathbf{x})} & > & 1 \nonumber \\
\exp(\boldsymbol{\beta}^{T} \mathbf{x}) & > & 1 \nonumber \\
\boldsymbol{\beta}^{T} \mathbf{x} & > & 0
\end{eqnarray} \] Therefore, we can see that two forms essentially learn the same classification boundary.

Logistic Loss

Since we establish the equivalence of two forms of Logistic Regression, it is convenient to use the second form as it can be explained by a general classification framework. Here, we assume \( y \) is the label of data and \( \mathbf{x} \) is a feature vector. The classification framework can be formalized as follows:
\[ \begin{equation}
\arg\min \sum_{i} L\Bigr(y_{i},f(\mathbf{x}_{i})\Bigl)
\end{equation}\]where \(f\) is a hypothesis function and \(L\) is loss function. For Logistic Regression, we have the following instantiation:
\[ \begin{eqnarray}
f(\mathbf{x}) &=& \boldsymbol{\beta}^{T} \mathbf{x} \nonumber \\
L\Bigr(y,f(\mathbf{x})\Bigl) &=& \log \Bigr( 1 + \exp(-y f(\mathbf{x})\Bigl)
\end{eqnarray}\]where \(y \in \{ \pm 1 \}\).

References

[1] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
[2] Tom M. Mitchell. Machine learning. McGraw Hill series in computer science. McGraw-Hill, 1997.
[3] Jason D. M. Rennie. Logistic Regression. http://people.csail.mit.edu/jrennie/writing, April 2003.

Rather than explaining the details of C++ 0x, I just post code here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

#include <random>
#include <iostream>
#include <string>
 
using namespace std;
 
int main()
{
  int n;
  double p, lambda, shape, mu, sigma;
  mt19937 eng(time(NULL));
 
  uniform_int_distribution<int> uniform_int(3,7);
  for(int i=0; i < 10; i++)
    cout << "[Uniform INT distribution]:" << uniform_int(eng) << endl;
  cout << endl;
 
  uniform_real_distribution<double> uniform_real(0.0,1.0);
  for(int i=0; i < 10; i++)
    cout << "[Uniform REAL distribution]:" << uniform_real(eng) << endl;
  cout << endl;
 
  n = 5;
  p = 0.3;
  binomial_distribution<int> binomial(n, p);
  for(int i=0; i < 10; i++)
    cout << "[Binomial distribution]:" << binomial(eng) << endl;
  cout << endl;
 
  lambda = 4.0;
  exponential_distribution<double> exponential(lambda);
  for(int i=0; i < 10; i++)
    cout << "[Exponential distribution]:" << exponential(eng) << endl;
  cout << endl;
 
  shape = 3.0;
  gamma_distribution<double> gamma(shape);
  for(int i=0; i < 10; i++)
    cout << "[Gamma distribution]:" << gamma(eng) << endl;
  cout << endl;
 
  p = 0.5;
  geometric_distribution<int> geometric(p);
  for(int i=0; i < 10; i++)
    cout << "[Geometric distribution]:" << geometric(eng) << endl;
  cout << endl;
 
  mu = 3.0; sigma = 4.0;
  normal_distribution<double> normal(mu, sigma);
  for(int i=0; i < 10; i++)
    cout << "[Gaussian distribution]:" << normal(eng) << endl;
  cout << endl;
 
  lambda = 7.0;
  poisson_distribution<int> poisson(lambda);
  for(int i=0; i < 10; i++)
    cout << "[Poission distribution]:" << poisson(eng) << endl;
  cout << endl;
 
  p = 0.6;
  bernoulli_distribution bernoulli(p);
  for(int i=0; i < 10; i++)
    cout << "[Bernoulli distribution]:" << bernoulli(eng) << endl;
  cout << endl;
 
  return (0);
}
}

Note, the code is very clean in the sense that you don’t need any extra libraries at all.

Please compile with:

g++ -std=gnu++0x

The GCC I used is GCC 4.6.1

Several references:
1. http://www.johndcook.com/test_TR1_random.html (sort of out-dated)
2. http://www.johndcook.com/cpp_TR1_random.html (sort of out-dated)
3. http://gcc.gnu.org/onlinedocs/libstdc++/manual/status.html#status.iso.200x (current status of C++ 0x in GCC)
4. http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt01ch03s02.html ( a list of header files of C++ in GCC)