Collaborative representation-based locality preserving projections for image classification

: Graph embedding has attracted much more research interests in dimensionality reduction. In this study, based on collaborative representation and graph embedding, the authors propose a new linear dimensionality reduction method called collaborative representation-based locality preserving projection (CRLPP). In the CRLPP, they assume that the similar samples should have similar reconstructions by collaborative representation and the similar reconstructions should also have the similar low-dimensional representations in the projected subspace. CRLPP first reconstructs each training sample using the collaborative representation of the other remaining training samples, and then designs the graph construction of all training samples, finally establishes the objective function of graph embedding using the collaborative reconstructions and the constructed graph. The proposed CRLPP can well preserve the intrinsic geometrical and discriminant structures of high-dimensional data in low-dimensional subspace. The effectiveness of the proposed is verified on several image datasets. The experimental results show that the proposed method outperforms the state-of-art dimensionality reduction.


Introduction
Dimensionality reduction has been widely studied and applied in the fields of computer vision and pattern recognition. Graph embedding is a famous type of dimensionality reduction and most of the dimensionality reduction methods can be unified into the general framework of graph embedding [1]. In this framework, the typical linear dimensionality reduction methods are principle component analysis (PCA) [2], linear discriminant analysis (LDA) [2] and locality preserving projections (LPP) [3]. Both PCA and LDA only consider the global geometrical structures of data, and they cannot perform well for the non-linear complex data. Unlike PCA and LDA, LPP is a manifold-based graph embedding method that can consider locality of data and can be suitable for discovering essential geometrical structures of linear and nonlinear data.
Since LPP was first introduced, its many variants are proposed because of the local geometry preserving property of LPP, such as in [4][5][6][7][8]. Among them, the locally discriminating projection (LDP) was proposed by using class information and similarities of samples for graph construction [4]. The locality-based discriminant neighbourhood embedding was proposed by maximising the margin between intra-class and inter-class samples [5]. The double adjacency graphs-based discriminant neighbourhood embedding was proposed by designing two adjacent graphs with class information [6]. The fast and orthogonal locality preserving projections (FOLPP) was proposed with orthogonal constraint [7]. The local similarity preserving function was designed for graph embedding [8]. We can see that these four methods above mainly utilise the geometrical and discriminant information of data to design graph constructions. In fact, the characteristic of many proposed extensions of LPP is that the different graph constructions are employed in graph embedding for dimensionality reduction.
Recently, many representation-based graph embedding methods are proposed by using the properties of preserving the potential geometry and natural discrimination of data through sparse representation [9] and collaborative representation [10], such as in [11][12][13][14][15][16]. Among these methods, the representative one is sparsity preserving projections (SPP) that was proposed by automatic graph construction using sparse representation [11]. Since SPP cannot consider the geometrical structures of data, its extensions were proposed by considering the locality through supervised sparse representation in [11][12][13][14] and by directly taking into account the geometry of data in [15,16]. In [16], sparsity and geometry preserving graph embedding (SGPGE) was proposed by integrating the sparse representation and geometry of data in graph embedding. In SGPGE, the geometrical structures of data are directly reflected by graph constructions. As the general one of sparse representation, collaborative representation [10] also has latent ability of preserving geometrical and discriminant information of data. Using the good property of collaborative representation, the collaborative representation-based projection (CRP) was proposed by using unsupervised collaborative graph construction [17]. To further discover the geometrical and discriminant structures of data, several extensions of CRP were currently developed by supervised collaborative graph constructions with class information [18][19][20].
As stated above, many variants of LPP cannot consider the geometrical distributions of each samples in graph constructions and the representation-based graph embedding methods hardly utilise the geometry of data directly with discrimination information of data. To overcome the limitations, in this paper we propose a collaborative representation-based locality preserving projection (CRLPP) method. In the proposed CRLPP, we assume that the similar samples could have the similar collaborative representations and their similar collaborative reconstructions also could have the similar projected representations in the embedded subspace. CRLPP first reconstructs each training samples through collaborative representation, and then constructs local adjacent graph using geometrical distributions of each training sample, finally the objective function of graph embedding using collaborative reconstructions and the graph construction is designed. The effectiveness of the proposed method is experimentally verified by comparing it with five competing graph embedding methods on four image datasets. Experimental results show that the proposed CRLPP performs better than the competing methods in the low-dimensional subspace for image classification.

Related works
In this section, we briefly describe two graph embedding methods including LDP and CRP that are very related to our proposed method. Here, the common used notations in the paper are first given. X = [x 1 , x 2 , …, x n ] ∈ R d × n is a set of original highdimensional data that has n training samples with d dimensionalities from C classes. The set of C classes is denoted as {1, 2, …, C}, the sample x i has class label c i ∈ {1, 2, …, C}. In linear dimensionality reduction, the original sample x i is projected into low-dimensional subspace as y i with y i = P T x i , where P = [p 1 , p 2 , …, p r ] ∈ R d × r and r ≪ d.

Locally discriminating projection
The LDP [4] is a supervised linear graph embedding method that fully uses class information to construct local adjacent graph. The adjacent graph is constructed by k-KNN to preserve local geometric structures of data. The adjacent weight W i j of edge between x i and x j is defined as follows: where α is a positive parameter, N k x i represents the set of knearest neighbours of x i . The objective function of LDP is defined as follows: Using the constraint p T PX DX T p = 1 and simple algebra operations, the objective function in (2) is rewritten as the following generalised eigenvalue problem: where L = D − W is a Laplacian matrix and D is a diagonal matrix with D ii + = ∑ j W i j . The r eigenvectors corresponding to the smallest r eigenvalues are chosen to make up the projection matrix.

CRP
The CRP is an unsupervised linear graph embedding method that uses collaborative representation to construct adjacent graph [17]. The adjacent graph in CRP is constructed by collaborative representation where s i = s i, 1 , …, s i, i − 1 , 0, s i, i + 1 , …, s i, n T ∈ R n and s i, i = 0 means that x i is represented by all the other training samples except x i . s i is regarded as the weight of edge from x i to x j . Then, the objective function of CRP is defined as Through some algebra operations, (5) with the constraint is reformulated as where S L = X I − S − S T + SS T X is a local scatter matrix and S T = ∑ i = 1 n x i − x¯x i − x¯T is the constraint that is a total scatter matrix of the data with the average x¯ of all the training samples. Finally, the projections P can be solved by the generalised eigenvalue problem The projections P is obtained as the matrix of the r eigenvectors corresponding to the r smallest eigenvalues of (7).

Proposed CRLPP
In this section, we first give the motivation of the proposed CRLPP and then describe CRLPP in details.

Motivation of CRLPP
As argued above, collaborative representation can discover geometrical and discriminant structures of data [10,17], but the collaborative representation-based graph embedding methods cannot directly employ the geometry of data [17][18][19][20]. Moreover, the many graph embedding methods that are the extensions of LPP cannot consider the geometrical distribution of each point sample of edges in the adjacent graph. To overcome the limitations in these competing graph embedding methods, in the proposed method we not only utilise collaborative representation but also employ the geometrical distribution of each point sample. In the proposed CRLPP, we assume that the similar training samples should have the similar collaborative reconstructions and the similar corresponding reconstructed ones should also have the similar projected ones in the low-dimensional subspace. Thus, in the proposed CRLPP, we first reconstruct each training sample using all the remaining training samples through collaborative representation, and then construct the discriminant and geometrical adjacent graphs using the class information and the geometrical distribution of each point samples, finally the objective function of graph embedding is designed by integrating the collaborative reconstructions and the graph construction.

Descriptions of CRLPP
First of all, using the collaborative representation each training sample x i is collaboratively reconstructed by (4) as where s i is the collaborative coefficient vector of x i and learned by (4). Through the collaborative reconstruction of each training samples, we expect that the good property of preserving inherent geometry and natural discrimination in collaborative representation can be preserved in the proposed method. Secondly, the local adjacent graph of data is constructed by k-NN. In the constructed adjacent graph, the class discrimination and the geometrical distribution of each point sample at the edges of graph are fully taken into account. The adjacent graph is denoted as G and the corresponding weight matrix is denoted as W. The weight W i j of the edge between x i and x j is defined as follows: J. Eng where t i k and t j k are the positive regulators of x i and x j , respectively.
t i k and t j k reflect the geometrical distributions of x i and x j in their regions of k-neighbourhood, respectively. The value of weight is composed of the local weight exp − ∥ x i − x j ∥ 2 /t i k and the intra-class enhanced discriminating weight 1 + exp − ∥ x i − x j ∥ 2 /t i k and it represents the geometrical and discriminant contribution of x i to x j . The value of weight indicates the geometrical and discriminant contribution of x j to x i . Just like the definition of the intra-class weight, the value of weight and it represents the geometrical and discriminant contribution of x i to x j . The value of weight represents the geometrical and discriminant contribution of x j to x i . Moreover, to make the weights reflect the geometry of each point, t i k and t j k represent the geometrical distributions of x i k and x j k , respectively. They are defined as and t j k = 1 As argued in [16], t i k can well reflect the local geometrical distribution of the point sample x i in the feature space. Meanwhile, to further reflect the geometrical distributions of each point in the adjacent graph, the weight W i j of each edge between x i k and x j k is the average of the weight from x i k to x j k and the weight from x j k to x i k . Thus, the designed local graph construction can discriminatively reflect the geometry of each samples of data.
Thirdly, to well preserve the local geometrical and discriminant structures of high-dimensional data and use the good property of collaborative representation in the low-dimensional subspace, we define the discriminative collaborative representation-based locality preserving objective function as follows: where y i = P T x¯i and x¯i = Xs i . Equation (12) can be further rewritten as where is a collaborative representation-based locality preserving scatter matrix. S k can be further reformulated as follows: where L = D − W is a Laplacian matrix, and D is a diagonal matrix that its entries are column sum of W, i.e. D ii = ∑ j W i j . Finally, with the orthogonal constraints P T P = I, the objective function of the proposed method can be reformulated as the following generalised eigenvalue problem: Through solving (15), the optimal transformation matrix P is composed by the r eigenvectors corresponding to the largest r eigenvalues. According to the descriptions above, we summarised the proposed CRLPP method in Algorithm 1. (i) Use (8) to collaboratively reconstruct each training sample x i by solving (4).
(ii) Use (9) to construct the adjacent graph W and compute collaborative representation-based locality preserving scatter matrix S k with (14).
(iii) Achieve the optimal transformation matrix P by solving (15).

Experiments
In this section, four image datasets are used in our experiments for validating the proposed CRLPP method. The four image dataset are ORL, COIL, PolyU_Palmprint and Folio. We compare the proposed method to the competing graph embedding methods that includes LDP SPP, CRP, FOLPP and SGPGE in terms of the recognition accuracy. In the experiments, the nearest neighbour classification method is adopted in the embedded subspace. On each dataset, the number l of the training samples are randomly chosen from each class and the remaining ones are the testing samples. This division one each dataset is carried out ten times, and the classification results are the average of the recognition accuracies over ten times. In addition, the value of nearest neighbours in FOLPP and LDP are set by the way in [21] and the values of the parameters in SGPGE are determined by the way in [16].

Date sets
The ORL dataset [http://www.cl.cam.ac.uk/research/dtg/attarchive/ facedatabase.html .] contains 40 subjects with 400 image faces, and each subject has image faces. Fig. 1a shows the example of face images from one subject on ORL. The COIL20 dataset [http:// www.cs.columbia.edu/CAVE/software/softlib/coil-20.php.] contains 1400 object images from 20 classes and each class has 70 images. Fig. 1b shows the example of images from one class on COIL20. The PolyU_Palmprint dataset [http:// www.comp.polyu.edu.hk/ biometrics/.] contains 1200 palm images from 60 classes and each class has 20 images. Fig. 1c shows the example of images from one class on PolyU_Palmprint. The Folio dataset [http://archive.ics.uci.edu/ml/datasets/Folio.] contains 640 images from 32 different types of leaves and each has 20 images. Fig. 1d shows the example of images from one class on Folio.

Experimental results
In this section, we first verify the classification performance with varying the numbers of the nearest neighbours k in the adjacent graph construction. The values of k is from 3 to 11 with a step of 1, and the numbers of training samples per class is set as l = 4, 5, 6, 7 on ORL, l = 20, 25, 30, 35 on COIL20, l = 6, 8, 10, 12 on PolyU_Palmprint and l = 6, 8, 10, 12 on Folio. The recognition accuracies of each competing method with varying the values of k are shown in Fig. 2. We can see from Fig. 2 that the parameter k has the significant influence on the proposed method in the graph construction. This means that the local geometry of data is very important in graph embedding. Furthermore, we can see that the proposed method achieves the better classification performance with more training samples. Using the experiments in Fig. 2, we further conduct comparative experiments to investigate the performance of the proposed method. In this experiment, the numbers of training samples per class are set as l = 7, k = 7 on ORL, l = 20, k = 11 on COIL20, l = 10, k = 5 on PolyU_Palmprint and l = 10, k = 9 on Folio. The classification performance of the proposed method with varying the dimensionalities of low-dimensional data is first studied. The values of dimensionalities are varied from 1 to 40 with a step of 1. The comparative classification results of each competing method are shown in Fig. 3. We can observe that the recognition accuracies of each method first quickly increase at the small values of dimensionalities, and then almost keep stable with the increase of dimensionalities. Moreover, we can clearly see that the proposed CRLPP always obtains better recognition accuracies than LDP, SPP, CRP, FOLPP and SGPGE among the range of dimensionalities. The maximal classification results of each method among the range of subspace on each image dataset are displayed in Table 1. Note that the best classification result among all the competing methods on each image dataset is indicated in bold-face. It is obvious that the proposed CRLPP performs best among all the competing methods on each dataset. Moreover, the best classification performance of the proposed method needs lower dimensionalities than the competing methods. Therefore, the experimental results demonstrate that the proposed CRLPP method is an effective and robust graph embedding method for dimensionality reduction.

Conclusion
Graph embedding is an effective technique in dimensionality reduction. To overcome the issue that the existing graph embedding methods cannot consider the geometrical distribution of each point sample in graph construction, we construct an informative local adjacent graph in graph embedding. Using the good property of preserving natural geometrical and discriminant information in collaborative representation, we propose a graph embedding method called CRLPP. The proposed method can well integrate the collaborative representation and the designed graph construction in the objective function of graph embedding. We conduct the experiments on four image datasets to compare the proposed CRLPP to the state-of-art graph embedding methods. The  effectiveness of the proposed method is demonstrated in image classification through the comparative experiments. The proposed method is a promising dimensionality reduction method that can well preserve the geometrical and discriminant structures of highdimensional data in the embedded subspace.