Tunable Kernel Model Based on Orthogonal Forward Selection with Tree Structure Search

Orthogonal Forward Selection based on Leave-One-Out Criteria (OFS-LOO) is recently proposed as an excellent tool for data modeling, which is capable of producing robust kernel model with tunable parameters. OFS-LOO adapts greedy scheme, which utilizes some global search algorithm to tune the kernel model term by term by minimizing LOO criteria. However, it is well known that the greedy algorithm only seeks the best performance in the current stage, and ignores its effect on the next stage. Nevertheless the selection of a particular regressor will surely have significant impact on the tuning of the regressor in the next stage. In this paper, a novel tree structure search is incorporated into the framework of OFS-LOO. The new method adopts repeated weighted boosting search (RWBS) algorithm. At each regressor, multiple optima are kept as the candidates of the parameters of the new regressor rather than only the best one is retained as the OFS-LOO does. This enhanced OFS-LOO provides a good compromise between an exhaustive search on all basis function parameters and a non-optimal a priori choice. The numerical results show that, compared to the traditional methods, the new approach can produce the kernel models with much more sparsity and better generality.