A Comparative Study on the Performance of New Ridge Estimators

Least square estimators in multiple linear regressions under multicollinearity become unstable as they produce large variance for the estimated regression coefficients. Hoerl and Kennard 1970, developed ridge estimators for cases of high degree of collinearity. In ridge estimation, the estimation of ridge parameter ( k ) is vital. In this article new methods for estimating ridge parameter are introduced. The performance of the proposed estimators is investigated through mean square errors (MSE). Monte-Carlo simulation technique indicated that the proposed estimators perform better than ordinary least squares (OLS) estimators as well as few other ridge estimators.


Consider the classical linear regression model
where X is a ( p n× ) matrix of non-stochastic regressors, β is a ( 1 × p ) vector of the unknown regression coefficients and u be a ( 1 × n ) vector of random disturbances such that For computational point of view X is normalized and y is expressed in deviations from mean.Ordinary least squares give the estimator for β as exist.OLS is an unbiased estimator.But when multicollinearity is present in the data, OLS estimator becomes unstable due to their large variance, which may lead to poor prediction.To overcome this condition, the most popular and commonly used estimator is the ridge estimator and it was first introduced by Hoerl and Kennard 1970.They defined the ordinary ridge estimator as where k > 0 is the ridge or shrinkage parameter.Ridge estimator is a biased estimator which is an alternative estimator to the OLS Estimator.Several methods are available in the literature to deal with the problem of multicollinearity.Some of the well known methods for choosing the ridge parameter are: Hoerl et al. 1975 Motivation for this paper is to study the performance of ridge estimators available in the literature and to suggest modified estimators when multicollinearity is present in the data.This article is restricted to deal multicollinearity problem.The proposed modified estimators are evaluated using Monte-Carlo simulation and compared in terms of ratio of average MSE (AMSE) of OLS over other existing ridge estimators.
For computational ease and for further discussion we express equation (1) in canonical form as where , where W be a ( p p × ) matrix such that its columns are normalized eigen vectors of X X ′, s λ i ' are the th j eigen value of X X ′.The ordinary least squares (OLS) estimator of y is then given by

Ordinary Ridge Estimator
By adding a biasing constant k to the th i element of the diagonal of the matrix Z Z′ (defined as in (3)), the ordinary ridge estimator (ORR) of γ can be written as where . From equations (3) and (4), we write The bias of R γ ˆis given by .) ( Therefore the bias of The mean square error of R γ ˆ is given by .

Proposed Estimators
Here we propose an estimator for the ridge parameter . ii) where max λ is the largest eigen value of .
, (Dorugade and Khashid 2010) is the variance inflation factor of the th j regressor. vi) The estimator HKB still works better in terms of MSE.The estimators DK and AD proposed by Dorugade and Khashid 2010 and Dorugade 2014 respectively, perform better than HKB when there exist a very high degree ( ) of collinearity among the predictors, which may not be realistic in real life situations.Simulation study indicates that when there is low or moderate or high degree of collinearity, the estimators KS, HMO and LW may tend to be unstable for the lower error variance.To overcome these we propose two modified estimators for determining ridge parameter , k and following Hoerl et al. 1975, the suggested modified estimators are defined as: Hadi-1988.Higher the value of m , higher is the degree of multicollinearity.If m is between 30-100 indicates moderate to strong correlation and if m is more than 100 suggests severe multicollinearity (Liu 2003).The simulation study indicate that suggested estimators defined as in ( 17) and ( 18) respectively perform better when data is suffering from low, moderate and high degree of collinearity.The suggested estimators SV 1 and SV 2 take little over bias than Hoerl and Kennard 1970 but they minimise the total variance.

The Results of Simulation Study
The performance of the estimators is studied through Monte-Carlo simulation.The performance of these estimators is investigated in the presence of low, moderate and high degree of multicollinearity.
where ij ξ is an independent standard normal pseudo-random number, ρ is specified such that 2 ρ is the degree of correlation between any two predictors.These predictor variables are standardized such that X X ′is in the correlation form and it is used to generate . To study the performance of the proposed estimators we have assumed various values for n as 10, 25, 50 and 100; variances of the residual term as 5, 10, 15, 25 and 100 and the degree of correlation ρ as 0.8, 0.9, 0.99 and 0.9999.Experiment is repeated 5000 times each and average mean square error (AMSE) is computed.Ridge estimates are computed by considering the different estimators of the ridge parameter ( k ) defined as in equations (11) to (18).Here we consider the process that leads to the maximum ratio of AMSE of OLS over AMSE of other ridge estimators to be the best in terms of MSE point of view.From tables -I and II, we observe that the performance of the suggested ridge estimators is better and comparable than the other estimators in almost all cases.However when there is a wide range of moderate or high degree of collinearity the estimator 2 SV k performs considerably better than all other estimators considered under study.If we observe carefully, the suggested estimators and as well the other estimators may vary little (under estimates on comparing with OLS) when the sample size ( n ) is large, degree of correlation is less and variance 2 σ of the error terms are small (Ref.table -1 for n =100; and error variance 2 σ = 5 and ρ = 0.8).
We observe the performance of different estimators from the Figures 1and 2. σ , for fixed n and .
ρ They indicate that overall performance of the suggested estimators is superior to other estimators considered under study.Further they clearly indicate that when predictors are either moderate or highly correlated the performance of the suggested estimators is satisfactory and perform similar to that of HKB and KD, and the performance of the estimators LW, AD, HMO, and KS are similar hence forming two groups in terms of their performance.A through study has to be done in this regard.

Conclusion
The ridge estimators studied in this article are computed for varied combinations of sample sizes ( n ), variance ( 2σ ) of the error term and degree of correlation ( ρ ) between the predictors.The suggested estimators are evaluated and compared with other ridge estimators.Experiment is repeated 5000 times each and average mean square errors (AMSEs) are computed.When there is wide range of degree of collinearity among the predictors the performance of the suggested estimators is satisfactory and slightly better over other ridge estimators considered in this articles.
condition number(Weisberg-1985, Chatterjee and Figure 1 is drawn for AMSE ratio against various values of sample size n , when ρ and 2 σ are fixed and Figure 2 is drawn for AMSE ratio against various values of error variance 2

Figures 1 :Figures 2 :
Figures 1: Visualization of performance of ridge estimators for various combinations of sample size (n) for fixed error variances (σ 2 ) and rho (ρ)] The results are obtained by generating a random matrix