Ex. 6.9

Explore the differences between the naive Bayes model (6.27) and a generalized additive logistic regression model, in terms of (a) model assumptions and (b) estimation. If all the variables \(X_k\) are discrete, what can you say about the corresponding GAM?

Soln. 6.9

Naive Bayes model is a generative classifier, which models the joint distribution \((X, Y)\) and then predicting the posterior probability \(P(y|x)\) with assumptions on \(P(X)\). Logistic regression is a discriminative classifier, which directly models the posterior probability \(P(y|x)\).

The naive Bayes model assumes that given a class \(G=j\), the features \(X_k\) are independent, while GAM only assumes the additivity of the model. The difference in the assumption leads to the difference in parameter estimation, though the two models have almost the same form. The additional assumption for naive Bayes dramatically simplify the estimation:

The individual class-conditional marginal densities can be estimated separately using 1-dimensional kernel density estimates.
When all the variables \(X_k\) are discrete, then appropriate histogram estimates can be used.

On the other hand, parameters are usually estimated by Newton-Raphson methods or backfitting algorithms (e.g., Algorithm 9.1 - 9.2 in the text) for GAM. When all the variables \(X_k\) are discrete (e.g., survey response data), it reduces to the so-called linear multinomial choice model and parameters can be fit by maximizing the conditional log-likelihood (Discrete choice analysis: theory and application to travel demand).