The Effect of Between Group Dependence on Measurement Equivalence/Invariance Tests by Brian Luis Perdomo A thesis submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Master of Science Auburn, Alabama December 12, 2011 Keywords: measurement equivalence, measurement invariance, structural equation modeling, confirmatory factor analysis, among-groups dependence, Monte Carlo simulation Copyright 2011 by Brian Luis Perdomo Approved by Daniel J. Svyantek, Chair, Professor of Psychology Bryan Edwards, Assistant Professor of Psychology Allison Jones-Farmer, Associate Professor of Business ii Abstract When using structural equation modeling methods, researchers should test for the presence of measurement equivalence/invariance (ME/I) before investigating substantive hypotheses (Bollen, 1989, p. 355). This study evaluated the robustness of the most common form of ME/I, metric invariance tests using multigroup confirmatory factor analyses, to violations of the independent groups assumption. Based on the analytically derived effect of such a violation on the likelihood ratio test, Jones-Farmer (2010) concluded that the results of the test would be unaffected. A Monte Carlo simulation was conducted to test her hypothesis and evaluate the robustness of other previously suggested ME/I tests that use CFI, SRMR, and RMSEA (Chen, 2007). As part of the study, the effect of among-groups dependence on the parameter estimates of factor loadings and their standard errors was also investigated. Among- groups dependence was found to have no practically significant effect on the results of the likelihood ratio test or the factor loading parameter estimates and standard errors, but other ME/I tests were impacted to varying degrees. iii Acknowledgments Since first arriving on campus to today, I have grown from an overgrown adolescent to a man, a husband, and a father. This was a long journey for me as a person and a graduate student and it would have been impossible to complete this on my own. I have to thank the members of my committee for their time, recommendations, and their seemingly unlimited patience with a student who looked like he would never get his act together. In particular, I would like to thank Dr. Allison Jones-Farmer whose previous work spawned this study. Without her expertise and confidence in my abilities, I probably would never have settled on a topic for my thesis or wrapped my head around its complexities. My family has always been the unshakable foundation upon which any of my own success was built. My parents have been my biggest fans for the past 30 years. Though I probably did not appreciate it at the time, the almost weekly question of ?How?s your thesis going?? from my mother served as a gentle push in the right direction. Her support has prevented me on several occasions from throwing in the towel and applying for a thrilling career at the nearest fast food establishment. My beautiful and patient wife, who for some inexplicable reason left the city-life of Miami to follow me to the comparatively rural confines of East- Central Alabama, has been my greatest cheerleader. She always knows what to say when I need to hear it. She is my best friend and I love her with all my heart. She has given me the greatest gift of all, my daughter Sophia, who has brought unfathomable joy into our lives and the additional inspiration I needed to get done. iv Table of Contents Abstract ........................................................................................................................................... ii Acknowledgments.......................................................................................................................... iii List of Tables ................................................................................................................................. vi List of Figures ............................................................................................................................... vii List of Abbreviations ..................................................................................................................... ix Introduction ......................................................................................................................................1 Approaches to Measurement Invariance Testing .................................................................3 Independent Groups Assumption in ME/I .........................................................................10 Need for Further Research .................................................................................................11 Method ...........................................................................................................................................14 Population Model Characteristics ......................................................................................14 Monte Carlo Simulation Design ........................................................................................16 Outcome Measures.............................................................................................................17 Results ............................................................................................................................................22 Factors Affecting Fit Indices .............................................................................................22 Noninvariance Conclusion Error Rates .............................................................................23 Invariance Conclusion Error Rates ....................................................................................25 Parameter Estimate Bias ....................................................................................................26 Parameter Estimate Standard Error Bias............................................................................27 Discussion ......................................................................................................................................28 v Limitations and Future Research .......................................................................................29 Application to Experimental Research ..............................................................................30 References ......................................................................................................................................47 Appendix A: Population Covariance Matrices ..............................................................................52 Appendix B: ANOVA Table of the Factors Affecting Fit Indices ................................................85 Appendix C: Error Rate Tables......................................................................................................90 Appendix D: Descriptive Statistics of the Parameter Estimates ....................................................97 Appendix E: Descriptive Statistics of the Parameter Estimate Standard Errors ..........................100 vi List of Tables Table 1: Summary of population covariance matrix manipulation conditions ..................................31 Table 2: Summary of model-implied covariance matrix characteristics used in analyses ................31 Table 3: Main effect of the number of noninvariant indicators on fit index values ..........................32 Table 4: Main effect of the latent factor correlation on fit index values ...........................................32 Table 5: Main effect of the uniqueness pair correlation on fit index values ......................................32 Table 6: Main effect of the test assumptions on fit index values.......................................................33 Table 7: Interaction effect of the test assumptions by the number of noninvariant indicators on fit index values .........................................................................................................................33 Table 8: Interaction effect of the test assumptions by the uniqueness pair correlation on fit index values ...................................................................................................................................33 Table 9: ANOVA results for invariant factor loading parameter estimates ......................................34 Table 10: ANOVA results for noninvariant factor loading parameter estimates ..............................34 Table 11: ANOVA results for invariant factor loading parameter estimate standard errors .............35 Table 12: ANOVA results for noninvariant factor loading parameter estimate standard errors .......35 vii List of Figures Figure 1: One-factor model fit to two groups assuming independence between groups. ..................36 Figure 2: One-factor model fit to two groups with a latent factor covariance between groups. .......36 Figure 3: One-factor model fit to two groups with covariance between corresponding uniquenesses across groups. ...............................................................................................37 Figure 4: One-factor model fit to two groups with covariance between latent factors and corresponding uniquenesses across groups. .......................................................................37 Figure 5: Comparison of the effect of test assumption on the noninvariance conclusion error rates for ??2 as a function of uniqueness pair correlation. ..........................................................38 Figure 6: Comparison of the effect of test assumption on the noninvariance conclusion error rates for ?SRMR as a function of uniqueness pair correlation. ..................................................38 Figure 7: Comparison of the effect of test assumption on the noninvariance conclusion error rates for ?RMSEA as a function of uniqueness pair correlation. ...............................................39 Figure 8: Comparison of the noninvariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are assumed. ..............39 Figure 9: Comparison of the noninvariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are not assumed. ........40 Figure 10: Comparison of the invariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are assumed. ............40 Figure 11: Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ??2 as a function of uniqueness pair correlation when independent groups are not assumed. ...............................................................................41 Figure 12: Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?CFI as a function of uniqueness pair correlation when independent groups are not assumed. ...............................................................................41 Figure 13: Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?SRMR as a function of uniqueness pair correlation when independent groups are not assumed. ...............................................................................42 viii Figure 14: Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?RMSEA as a function of uniqueness pair correlation when independent groups are not assumed. ...............................................................................42 Figure 15: Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for Chen's (2007) rule as a function of uniqueness pair correlation when independent groups are not assumed. ...................................................43 Figure 16: Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have no correlation. .........................................................43 Figure 17: Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a low correlation. ....................................................44 Figure 18: Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a moderate correlation. ...........................................44 Figure 19: Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a high correlation. ...................................................45 Figure 20: Comparison of the sampling distributions of invariant factor loadings. ..........................45 Figure 21: Comparison of the sampling distributions of noninvariant factor loadings. ....................46 ix List of Abbreviations CFA Confirmatory Factor Analysis CFI Comparative Fit Index CTT Classical Test Theory DIF Differential Item Functioning IRT Item Response Theory MCFA Multigroup Confirmatory Factor Analysis ME/I Measurement Equivalence/Invariance ML Maximum Likelihood RMSEA Root Mean Square Error of Approximation SRMR Standardized Root Mean Square Residual 1 Introduction In the organizational sciences, a wide variety of research questions are aimed at differences among groups (Vandenberg & Lance, 2000). Cross-cultural studies and selection discrimination research, in particular, rely heavily on group comparisons. Originally, it was assumed that the only thing required to investigate this between-group variation was simply to compute the difference between the observed scores of the groups (Cronbach & Furby, 1970). Since then, Cronbach (1992) and Edwards (2002) have put forth several potentially problematic issues in the use of difference scores. When comparing phenomena across groups, one must ensure that the measures of those phenomena are not only reliable and valid but also that they are measured on the same scale for all groups. It is essential that the scale is understood and used similarly by the members of the various groups (Meade & Lautenschlager, 2004). Two measurements of the same phenomenon must be psychometrically equivalent to make substantive comparisons across groups or over time (Horn & McArdle, 1992). Measurement equivalence or invariance (ME/I) testing addresses the question of whether or not different groups respond to a set of items similarly. In the context of cross-cultural research, Riordan and Vandenberg (1994) point out that a lack of ME/I could invalidate any substantive conclusions drawn from any group comparisons. If two groups view a particular scale differently, as evidenced by a lack of ME/I, then any inferences drawn regarding mean differences on the latent factor measured by the scale could be misleading. A common assumption of researchers who conduct ME/I tests is that the groups are inde- pendent or that the cases can be matched across groups (Jones-Farmer, 2010). In some cases, the researcher?s ability to match observations across samples can be hindered through anonymity restrictions, attrition, errors in data collection or incomplete archival data. In these cases, 2 researchers often analyze the data as though it were independent (Bedeian & Feild, 2002). Although the independence assumption is untenable, it may still be possible to use the data from these studies to answer questions related to ME/I. The purpose of this study is to investigate the repercussions of violating the independent groups assumption in ME/I testing using a Monte Carlo simulation. The importance of ME/I testing extends beyond academic research. Tests that are used for personnel selection decisions in organizations and admissions decisions in educational settings must be tested for ME/I. Bollen (1989, p. 355) warns that decisions made for individuals in the absence of measurement invariance can be seriously flawed. Observed score differences can reflect not only true group mean differences, but also differences in the relation between the construct and the observed score that are not equivalent across groups (Raju, Lafitte, & Byrne, 2002). These measurement processes must employ procedures to determine the influence of variables, such as group membership, that should be irrelevant to the construct under study (Messick, 1989). The theoretical foundations for ME/I testing were developed over 30 years ago (e.g., Meredith, 1964 and Mulaik, 1972) and the literature on ME/I has spanned several disciplines such as gerontology, education, developmental psychology, marketing, and criminology (Vandenberg & Lance, 2000). More recently, the topic has been re-energized due to several factors. The increased computational capacity on personal computers as well as software packages?LISREL, EQS, Mplus, and Amos?that are more readily available and accessible to users with less programming experience. An increase in the number of articles and conference papers on the subject of ME/I has led to broader recognition of its importance among researchers 3 in several disciplines. There is also an increased understanding of the procedures required to establish ME/I (e.g., Byrne, Shavelson, & Muth?n, 1989; Vandenberg & Lance, 2000). Approaches to Measurement Invariance Testing Many methods for detecting ME/I have been proposed. The most prominent division is between those methods that are based on classical test theory (CTT) and those derived from item response theory (IRT; Stark, Chernyshenko, & Drasgow, 2006). The IRT-based approaches to ME/I testing were originally developed for studying differential item functioning (DIF), or test bias. In CTT-based approaches, multisample confirmatory factor analysis (MCFA) is used to test a priori theories of model structure across groups (Alwin & Jackson, 1981; McGaw & J?reskog, 1971) or across time (Mantzicopoulos, French, & Maller, 2004). Several studies have compared these two approaches (e.g., Raju, Lafitte, & Byrne, 2002; Stark, Chernyshenko, & Drasgow, 2006) and found that, despite some idiosyncrasies, they generally provide similar results though each method has certain strengths and weaknesses. For instance, IRT methods perform better for measures with dichotomous items while the MCFA procedures appear to be better suited to scales with polytomous items. Also, the use of multiple group comparison procedures for IRT-based methods is generally confined to unidimensional scales. MCFA procedures can be used to compare multiple groups on unidimensional and multidimensional scales. This study only considers the MCFA procedure of ME/I testing as the current literature suggests it to be more broadly applicable for ME/I testing. Confirmatory factor analysis approach. In CTT, an individual?s observed score on each item in a scale is viewed as the sum of the individual?s true score on the construct the scale purports to measure and an error component and is represented as 4 (1) where, for the ith individual, is the observed score on the jth item, is the true score, and is the nonsystematic error of jth item. Theoretically, this error term is normally-distributed with a mean of zero and variance, . It is also assumed that the errors are uncorrelated with the true score. This assumption allows the variance of the observed scores to be similarly decomposed into its components: (2) In the MCFA framework, the true score is conceptualized as a latent factor (?) and the observed item scores (X) are related linearly to the latent factors. For the gth group, the relationship is represented as (3) where, refers to the vector of observed scores for the kth scale item, refers to the matrix of regression slopes relating to the latent factor ( ), refers to the vector of regression intercepts, and refers to the vector of unique factors, or errors. Equation 3 is a more general form of the most common covariance structure analysis equation. Normally, the intercepts, , are assumed to be zero and are not estimated (J?reskog & S?rbom, 1996, p. 297). Assuming that the errors and latent factors are independent, the covariance equation that follows from Equation 3 is (4) where is the matrix of variances and covariances among the items in the gth population (group), is the matrix of the items? factor loadings on the latent factors, contains the variances and covariances among the latent factors, and is the matrix of the variances and 5 covariances among the errors. Equation 4 is the fundamental covariance equation in factor analysis (J?reskog & S?rbom, 1996, p. 3). Traditionally, the likelihood ratio test for G groups evaluates the hypotheses, ( ) vs. ( ). The unknown population covariance matrix is of the form [ ] (5) where ?g is the unknown population covariance matrix of the gth group as defined in Equation 4, and 0 represents the k x k matrix of zeros. In the likelihood ratio test, ? is estimated using the sample covariance matrix S of the observed random variables of the form [ ]. (6) This estimator of the population covariance matrix is then compared to the model-implied covariance matrix, ( ) determined by the investigator that relates the observed random variables to the latent factors. The model-implied covariance matrix is of the form, ( ) [ ( ) ( ) ( ) ] (7) It consists of a block-diagonal matrix with each group?s model-implied covariance matrix on the diagonal and k x k zero matrices on the offdiagonal blocks. For group, g, the model-implied covariance matrix is of the form ( ) , (8) 6 which is similar to Equation 4 where the covariance matrix is a function of the vector of fixed and free parameters, ?g, rather than being based on the true population covariance matrix, The presence of zero matrices in all offdiagonal submatrices reflects the independent groups assumption of the likelihood ratio test. The likelihood ratio test statistic, ( ) ( ( ?)), (9) is based on the maximum likelihood discrepancy function given by ( ( )) | ( )| ( ( )) | | , (10) evaluated at its minimum value, ? Assuming the model is specified correctly T is asymptotically-distributed as a central chi-square random variable with degrees of freedom, ( ) , where s is the number of free parameters in Vandenberg and Lance (2000) used these equations to illustrate the research questions related to ME/I that can be tested using the MCFA framework. After reviewing the previously published research on ME/I testing, Vandenberg and Lance provided their recommendations for which tests to conduct and the sequence of those tests. The first five tests all concern the psychometric properties of the measurement scales. The last three tests have to do with between- group differences in latent means, variances, and covariances. Vandenberg and Lance suggest that these tests are used to establish measurement invariance and structural invariance, respectively. Little (1997) referred to these as Category 1 and Category 2 invariance, respectively, and suggested that, in general, Category 1 invariance is a prerequisite for the unambiguous interpretation of Category 2 differences. Tests of Category 2, or structural, invariance are usually of substantive research interest (Cheung & Rensvold, 2002). These tests are implemented by modifying the vector of the model-implied covariance matrix. The 7 following sections will only describe the tests of measurement invariance as they are more pertinent to the current study. Invariant covariance. Although only 26% of the studies reviewed by Vandenberg and Lance (2000) conducted the test of invariant covariance matrices, , the test is recommended as an initial omnibus ME/I test. The research question is addressed by testing the equality of the covariance matrices of the samples, using the likelihood ratio test statistic and/or other goodness-of-fit indices. A failure to reject the null hypothesis is interpreted as evidence of overall ME/I across groups and, thus, no further testing is required. Rejection of the null hypothesis, however, provides no information regarding the source of measurement nonequivalence. Byrne (1989, p. 126) states that the rejection of the null hypothesis in this test should lead to the testing of a ?series of increasingly restrictive hypotheses in order to identify the source of nonequivalence.? Configural invariance. The least restrictive ME/I hypothesis that should be tested first is the equivalence of the conceptualized relationships between the observed variables and the latent variables across groups, or configural invariance (Bollen, 1989, p. 358). To ensure that two scales measure the same constructs, it is essential that they have the same dimensionality and that the items that load on each factor be the same for any group comparisons to be made (Horn & McArdle, 1992). For example, a self-esteem measure would lack configural invariance if it was found that the items load on a single factor for men and three factors for women. Furthermore, even if the measure exhibits an equal number of latent factors, the items that load on each factor should be the same across groups. This test is conducted by simultaneously testing the fit of the same a priori measurement model for each group. If the null hypothesis is rejected, then no further tests of 8 ME/I or other group differences can be done. If the model fits well for each group then there is evidence that the groups exhibit configural invariance. This suggests that the groups use the same conceptual frame of reference for the underlying constructs and the results warrant more stringent tests of ME/I (Vandenberg & Lance, 2000). In several studies, this test is referred to as the ?baseline? model because all further tests are nested within the test of configural invariance (e.g., Bagozzi & Edwards, 1998; Marsh, 1994; Reise, Widaman, & Pugh, 1993). Metric invariance. The null hypothesis of the test of metric invariance is (Horn & McArdle, 1992). The test looks at the relationships between the scale items and their underlying constructs, more specifically the factor loadings of the scale items ( ). The hypothesis is most commonly tested using the likelihood ratio test, or difference test in which the difference between the statistics for a model in which the loadings are constrained to be equal across groups and one in which they are unconstrained is computed. The failure to reject the null hypothesis suggests that the items are measured on the same scale by each group. There is some debate as to what can be done if the null hypothesis is rejected (Vandenberg & Lance, 2000). Some researchers advise that no further tests of ME/I should be conducted after the rejection of the null hypothesis for the metric invariance test (e.g., Bollen, 1989; Millsap & Hartog, 1988). Others advocate tests of partial metric invariance in which items that exhibit ME/I are held equal across groups and the rest are free to be estimated separately for each group (see Byrne, 1991; Byrne & Baron, 1994; Byrne, Baron, & Campbell, 1993, 1994; Finch et al., 1989; Marsh & Hocevar, 1985). 9 Scalar invariance. Scalar invariance refers to the equivalence of the item intercepts across groups, . It is a prerequisite for the comparison of the means of the latent variables because, if both metric and scalar invariance exist, it suggests that the items are measured on the same scale across groups. Without both forms of invariance, between-group differences in latent means can be confounded with differences in the scale and origin of the latent variable (Cheung & Rensvold, 2002). The presence of scalar invariance is tested using the likelihood ratio test while constraining the vector of item intercepts to be equal across groups. If the null hypothesis is rejected, Byrne et al., (1989) also suggested that latent means could be compared under the condition of partial scalar invariance, but, as with metric invariance, some authors do not recommend this approach (Vandenberg & Lance, 2000) Residual invariance. The uniqueness, or residual, variance is the portion of the item variance not attributable to its associated latent variable. Testing for residual invariance is used to determine if the scale items measure the latent construct with an equivalent amount of measurement error (Cheung & Rensvold, 2002). Residual invariance is investigated using the likelihood ratio test while constraining the item uniquenesses across groups, . If residual invariance is not found, then any latent mean differences between groups could potentially be confounded due to measurement error. Most researchers have treated residual invariance as a test for invariant indicator reliabilities across groups (e.g., Schmitt, Pulakos, & Lieblein, 1984), but Vandenberg and Lance (2000) warn that this is only true if the factor variances are invariant across groups. 10 Independent Groups Assumption in ME/I Traditionally, the ME/I tests have been based on the assumption of independent groups. Jones-Farmer (2010) analytically derived the effect of among-groups dependence on the likelihood ratio test of ME/I. In classical linear models, tests which fail to account for group dependence lead to inflated mean square error estimates and Type II error. In the context of ME/I tests, a Type II error means that the researcher has concluded mistakenly that ME/I exists and would then pursue potentially spurious substantive group comparisons. In the case of among-groups dependence, the offdiagonal blocks of the population covariance matrix (Equation 5) would not be zero matrices. Instead, they would be of the form [ ] (11) where ?ij represents the k x k covariance matrix between xi and xj. Similarly, to incorporate the among-groups dependence, the model-implied covariance matrix would be ( ) [ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ] (12) where ?ij(?ij) represents the k x k model-implied covariance matrix between xi and xj and the sample covariance matrix would be [ ] (13) where Sij represents the k x k covariance matrix between xi and xj. Jones-Farmer (2010) concluded that the misspecification of the model due to assuming independent groups when some correlation exists would have no effect on the difference test. She showed that the 11 nested models used to calculate the test statistics are equally misspecified and by taking the difference between the two test statistics any effect on model fit due to model misspecification is canceled out. Thus, there is no correction required for the difference test statistic. Need for Further Research In their literature review, Vandenberg and Lance (2000) emphasized the importance of establishing ME/I prior to making substantive comparisons across groups. Vandenberg (2002) called for further methodological research to be done in this area. He stated that practitioners should not assume ME/I procedures are accurate in all circumstances and urges researchers to continue to evaluate these procedures in different contexts and conditions. In response to that call for research, several studies have investigated various aspects of ME/I tests. For instance, a few studies have looked at the analysis of ordinal data (Flora & Curran, 2004; Lubke & Muth?n, 2004; Millsap & Yun-Tein, 2004). Also, several studies have reported Monte Carlo simulations of different aspects of the MCFA approach to ME/I testing. Cheung and Rensvold (2002) investigated the use of 20 different goodness-of-fit indices in addition to the difference test for investigating ME/I. They suggest the use of ?CFI to supplement conclusions drawn to the difference test. All investigations of ME/I in the Cheung and Rensvold (2002) study were done under the assumptions of multivariate normality and independent groups and only using maximum likelihood (ML) estimation. Also, they only examined Type I error and suggested that future research should examine power as well. Meade and Lautenschlager (2004) found that ME/I tests perform well under ideal conditions (i.e., large sample sizes, a sufficient number of indicators, and moderate communalities, and independent groups). French and Finch (2006) investigated the power and Type I error rates of the difference test under a variety of 12 conditions, including various sample sizes, number of factors, number of indicators per factor, and distributions of observed variables. They found that the difference test adequately controls Type I error and provides high power when used with ML estimation, multivariate normal variables, and independent groups. Similarly, Butts, Vandenberg, and Williams (2006) evaluated the effect of common method variance on the power and Type I error rates of the difference test and ?CFI under varying conditions of configural and metric invariance. Chen (2007) studied the ability of several goodness-of-fit indices to identify various levels of measurement invariance and recommended the use of a combination of the differences in standardized root mean square residual (?SRMR), comparative fit index (?CFI), and the root mean square estimate of approximation (?RMSEA) and provided cutoff scores for each. In a Monte Carlo simulation study of the precision of factor loadings in ME/I tests, Meade and Bauer (2007) found that no single rule of thumb can be used to determine the appropriate sample size for the tests. All of these previous studies have assumed that the groups are independent. The independent groups assumption is not always tenable. For example, it is common for researchers to sample subjects from related populations such as students from different classes in the same college, employees from different companies within the same industry, etc. Because these subjects may be similar on many factors such as demographics, geographic location, socioeconomic status, cognitive ability, age, etc., the assumption of independence across groups may be incorrect. The independence assumption can also be violated when conducting a longitudinal study of subjects with repeated observations on the same sample of subjects. In some cases, the researcher?s ability to match observations across samples can be hindered through anonymity restrictions, attrition, errors in data collection or incomplete archival data (Jones-Farmer, 2010). In these cases, researchers often analyze the data as though it were 13 independent (Bedeian & Feild, 2002). Although the independence assumption is not valid, it may still be possible to use the data from these studies to answer questions related to ME/I. To date, two studies have addressed the issues related to violating the independent groups assumption in tests when comparing groups in a CFA framework. As mentioned previously, Jones-Farmer (2010) analytically investigated the effects of among-groups dependence on ME/I testing in confirmatory factor analyses. Papadopoulos and Amemiya (2005) conducted an asymptotic robustness study for a general structural equation model and developed corrections to the asymptotic standard errors of the parameters of the model. As part of their theoretical analysis, they investigated cases in which the groups may be correlated. They provide a small simulation and an empirical example to support their theoretical results. Papadopoulos and Amemiya did not, however, speculate as to what effect these corrections would have on tests of ME/I. This simulation study will investigate the findings of the Jones-Farmer (2010) study and evaluate the standard errors of the parameter estimates as suggested by Papadopoulos and Amemiya (2005) and their application to tests of ME/I. Using a Monte Carlo simulation, data will be simulated with known population covariance matrices that exhibit a lack of ME/I and correlation between groups. The current study will investigate the effect of among-groups dependence on the tests of ME/I. The study will also expand the literature by examining the performance of the use of other goodness-of-fit indices as measures of ME/I as suggested by Chen (2007) and by examining the effect of among-groups correlation on the Type I and Type II error rates of these decision rules. The results of this study will be used to develop recommendations for researchers that conduct multigroup analyses using structural equation models with samples whose independence is unclear. 14 Method A two-step procedure was used to generate the data tested in this study using the Monte Carlo function in Mplus 5.2 (Muth?n & Muth?n, 2007) and population models exhibiting the properties of all combinations of manipulation conditions. The tests were performed using the MCFA approach and ML estimation. Because Vandenberg and Lance (2000) found in their review of existing ME/I studies that metric invariance was the most commonly used ME/I test, this study was constrained to tests of metric invariance. In future studies, other ME/I tests should be studied as well. The resulting fit statistics were examined to determine the rates of incorrect conclusions based on the ME/I decision rules for the likelihood ratio test and the fit indices. Population Model Characteristics All population models consisted of two groups, each with one latent factor and five manifest indicators. This is intended to simulate a short survey consisting of five items that measure a single construct for two groups. The variance of the latent factors was constrained to equal one in both groups for model identification purposes and to allow all indicators the possibility of being noninvariant. Figure 1 presents the graphical representation of the general population model when the groups are assumed to be independent. Number of noninvariant items. In a given scale, it is possible for any number of items to exhibit a lack of ME/I. In the current study, models were tested in which one to five indicators exhibit a lack of ME/I across groups. These conditions represent situations in which the psychometric properties of 20% to 100% of the scale items differ across groups. A control condition was also included in which no indicators exhibit a lack of ME/I. Factor loadings in Group 2 that exhibit a lack of ME/I were 15 fixed to 0.20 lower than Group 1 loadings. The uniqueness of any noninvariant indicator was adjusted so that the total variance of each indicator, whether invariant or not, was equal to 1.14. Group correlation. The correlation between Group 1 and Group 2 was manipulated by adjusting the degree to which either the latent factors (Figure 2), the uniquenesses of the indicators (Figure 3), or both (Figure 4) covary across groups. Correlation between groups were set to either 0.00 (Independent), 0.10 (Low), 0.30 (Moderate), or 0.50 (High). The different sources of potential group correlations were evaluated separately because they have different implications for a particular sample. A strong correlation between the latent factors of the two groups would indicate that all indicators are correlated to some degree across groups depending on how strongly the indicators load on that factor. Ultimately, a strong correlation between latent factors would most likely occur in situations in which the indicators are invariant across groups. By definition, uniqueness refers to the amount of indicator variance that is not accounted for by the latent factor. A high correlation among uniqueness pairs is not necessarily indicative of measurement invariance and, in fact, would have unpredictable and potentially detrimental effects on model fit and ME/I test results. Model parameter simulation. The variance of the latent factors of the population models for Group 1 and Group 2 across all conditions was constrained to be fixed at 1.00 for identification purposes. All factor loadings of indicators in Group 1 were fixed at 0.80. The factor loading of the indicators in Group 2 were dependent on the manipulation conditions as discussed previously. Each experimental condition was replicated 5,000 times, with the number of replicates chosen so that 16 the standard error of the Type I and Type II error rates were below 0.005 for each simulated case. The population covariance matrices used to generate each sample is provided in Appendix A. Monte Carlo Simulation Design In the first step of the Monte Carlo simulation, Mplus 5.2 (Muth?n & Muth?n, 2007) was used to generate 5000 samples for each of the 96 population models. Sample sizes were held constant at 500 for both groups across all replications. This sample size was chosen so that it would be sufficient to suitably recover population parameters and is representative of samples available to many researchers and practitioners conducting these types of assessments (Meade & Lautenschlager, 2004). A summary of the simulated data conditions that are described in the following sections is provided in Table 1. In the second step, the generated sample data was used to test the fit of four different models as described in Table 2. Models 1 and 2 represent the traditional ME/I test of metric invariance under the assumption of independent groups using the sample and model-implied covariance matrices from Equations 6 and 7. In Model 1, the fit of the two group model is evaluated when the factor loadings are free to be estimated based on the sample data with no constraints. In Model 2, the fit of the two group model is evaluated when each factor loading is constrained to be equal across groups. If metric invariance exists, then the fit indices should not differ significantly. Models 3 and 4 represent a test of ME/I that removes the independent groups assumption using Equations 11 and 12 as the model-implied and sample covariance matrices. 17 Outcome Measures Fit indices. The likelihood ratio test statistic ( ) and several goodness-of-fit indices were recorded to examine the ability of each index to identify measurement invariance across the various conditions. The other indices, ?CFI, ?SRMR, and ?RMSEA, included in the current study are based on the recommendations of Chen (2007). Furthermore, Chen recommended a decision rule that uses a combination of these fit indices which was also evaluated. Chi-square. The test statistic of the likelihood ratio test is based on the maximum likelihood discrepancy function evaluated at its minimum (Equation 9). In the analyses in which the groups are assumed to be independent, the minimized maximum likelihood discrepancy function is of the form ( ( ?)) ?( )( | ( ? )| ( ( ? )) | | ) (14) when ? = ?(?), the likelihood ratio test statistic is asymptotically distributed as a central chi- square random variable with degrees of freedom equal to ( ) , where s is the number of freely estimated parameters in ?. When the independent groups assumption is removed, the discrepancy function is of the form ( ( ?)) | ( ?)| ( ( ?)) | | (15) where S is defined in Equation 13 and ( ?) is defined in Equation 12 evaluated at its minimum. When ? = ?(?), the likelihood ratio test statistic is asymptotically distributed as a central chi- 18 square random variable with degrees of freedom equal to ( ) , where s is the number of freely estimated parameters in ? (Jones-Farmer, 2010). Comparative fit index. The comparative fit index (CFI) is an incremental fit index widely used to evaluate the fit of structural equation models. CFI assesses the relative improvement of the fit of the researcher?s model as compared to a baseline model which is typically the null model. CFI is defined as ? ? (16) where ? and ? are estimators of the noncentrality parameters of a noncentral chi-square distribution for the researcher?s model and baseline model, respectively. The noncentrality parameter estimator is defined by ? ( ). (17) Chen (2007) recommended a change of CFI ? -0.01 would indicate the presence of noninvariance. This cutoff was proposed for ME/I tests under the independent groups assumption. Standardized root mean square residual. The standardized root mean square residual (SRMR) is an absolute fit index based on the differences between observed and model-implied covariance defined by ? ??[( ) ( )? ] ( ) (18) 19 where sij is the observed covariance, ?ij is the model-implied covariance, and sii and sjj are the observed standard deviations. Chen (2007) recommended a change of SRMR ? 0.03 would indicate the presence of noninvariance under the independent groups assumption. Root mean square error of approximation. The root mean square error of approximation (RMSEA) seeks to evaluate the lack of fit of the researcher?s model to the population covariance matrix. RMSEA is referred to as a parsimony-adjusted index because it includes a correction for model complexity and is defined by ? ? ( ) (19) where ? is the noncentrality parameter estimator of a noncentral chi-square distribution for the researcher?s model as defined in Equation 17. Chen (2007) recommended a change of RMSEA ? 0.015 would indicate the presence of noninvariance under the independent groups assumption. Chen's rule. Chen (2007) recommended that a combination of these fit indices should be used to make ME/I conclusions. When group sample sizes are equal and total sample size is over 300, Chen suggested that changes in CFI ? -0.01, supplemented by either a change in SRMR ? 0.03 or RMSEA ? 0.015 indicates noninvariance. Noninvariance conclusion error rates. Using the samples generated from population covariance matrices that have no invariant indicators, the number of incorrect noninvariance conclusions based on decision rules for each fit index was recorded. These refer to the replications in which a researcher would determine that 20 the indicators are not invariant between the two groups based on the data. For the likelihood ratio test, a significance level of 0.05 or less was recorded as a rejection of the null hypothesis. Noninvariance conclusion error rates for the likelihood ratio test are synonymous with Type I error, but the more general term is used in this instance to encompass the results of the fit index decision rules which are not true null hypothesis tests. It is important to note that an incorrect noninvariance conclusion in the context of ME/I tests suggests that the items are not invariant and that no further testing should be conducted. Upon arriving at this result, a researcher would not investigate any substantive conclusions that may have been drawn otherwise. Invariance conclusion error rates. Using the samples generated from the population covariance matrices that contain noninvariant indicators, the number of incorrect invariance conclusions was recorded using the same decision rules as described previously. In this case, incorrect identifications are those replications in which one or more noninvariant items exist and were not identified by the likelihood ratio test and/or the fit index decision rules. Invariance conclusion error rates for the likelihood ratio test are synonymous with Type II error. As with noninvariance conclusion error rates, the more general term, invariance conclusion error rate, is used to include conclusions drawn from the fit index decision rules as well as the likelihood ratio test. These types of errors can be the most problematic because the researcher would test further substantive hypotheses under the false conclusion that ME/I exists. Any findings based on these subsequent tests would be untenable. Factor loading biases and standard errors. Among-groups dependence could potentially bias the estimates of the factor loadings. Papadopoulos and Amemiya (2005) highlight the potential effects that among-groups 21 dependence can have on the standard errors of the parameter estimates as well. Chen (2007) noted that the standard errors of the fit indices were only affected by sample size. However, this conclusion was under the assumption of independent groups. In the current study, the parameter estimates and their standard errors were recorded to explore any potential effects of ME/I and between-groups dependence. 22 Results Factors Affecting Fit Indices To explore factors that affected changes in goodness of fit indexes, a 6 (Number of Noninvariant Items: 0 - 5) x 4 (Between Groups Factor Correlation: 0.0, 0.1, 0.3, and 0.5) x 4 (Corresponding Uniqueness Pair Correlation: 0.0, 0.1, 0.3, and 0.5) x 2 (Independent Groups Assumed vs Not Assumed) analysis of variance (ANOVA) on the test statistic and each fit index was conducted. Due to the large sample size, many of the main factors and interactions in the analyses were statistically significant. It is much more useful in this context to examine the practical significance of each of the remaining factors and interactions on the outcomes of interest. Appendix C includes the full results for each of these ANOVAs. Number of noninvariant indicators. The number of manifest indicators that were noninvariant between groups can be thought of as the effect size of the ME/I test. As the effect size increases, the power of the ME/I test will increase as well. The number of noninvariant indicators significantly affected all the fit indices (Table 3). Latent factor correlation. Latent factor correlation between the groups was statistically significant, but it had no practical effect on any of the fit indices (Table 4). These results were most likely due to the very large sample size and will be excluded from further discussions of results. Uniqueness pair correlation. The main effect of the correlation between corresponding pairs of uniquenesses between groups was statistically significantly for all fit indices (Table 5). However, the practical 23 significance of this effect was low. This may be due, in part, to an interaction effect between the test assumptions and the uniqueness pair correlation. Test assumptions. The main effect of the test assumptions under which the ME/I tests were conducted was statistically significantly for all fit indices (Table 6). However, the practical significance of this effect was negligible for both ??2 and ?CFI. The test assumptions account for a much larger portion of variance in ?SRMR and ?RMSEA. In the case of ?RMSEA, this effect is even larger than that of the number of noninvariant items, the actual effect of interest in ME/I tests. Interaction effects. Although most of the interaction effects were statistically significant, only two are noteworthy in terms of practical significance. The effect of the interaction between test assumptions and the number of noninvariant items was statistically significantly for all fit indices (Table 7). Again, there was virtually no practical significance of this effect for both ??2 and ?CFI, but the interaction did account for a larger portion of variance in ?SRMR and ?RMSEA. The effect of the interaction between test assumptions and the uniqueness pair correlation was statistically significantly for all fit indices (Table 8). There was no practical significance of this effect for ??2, ?CFI, and ?SRMR, but the interaction did account for a larger portion of variance in ?RMSEA. Noninvariance Conclusion Error Rates In the current study, a population model in which all of the indicators were invariant was used to evaluate the percentage of the ME/I tests that incorrectly identify these indicators as noninvariant. Incorrect conclusions of this type would result in a researcher believing that the factor loadings are not equal across groups and may not investigate further substantive 24 hypotheses. The noninvariance conclusion error rates for the test (Figure 5) show that neither the independent groups assumption nor the actual presence of group correlation have a practically significant influence on the ME/I test results. The noninvariance conclusion error rates are all between 0.037 and 0.048. When using the ?CFI and Chen?s (2007) ME/I decision rules, the noninvariance conclusion error rates for all conditions was 0.000. The noninvariance conclusion error rates for the ?SRMR (Figure 6) and ?RMSEA (Figure 7) ME/I decision rules are larger in the analyses in which independent groups are assumed to exist regardless of group correlation. Also, for both decision rules, as the group correlation increases the noninvariance conclusion error rates decrease slightly. The noninvariance conclusion error rates for ?SRMR when groups are assumed to be independent are between 0.258 and 0.271. When group are not assumed to be independent, the error rates are between 0.002 and 0.037. This disparity exists in the noninvariance conclusion error rates of ?RMSEA, but the rates are higher when the groups are assumed to be independent. In this case, the noninvariance conclusion error rates for ?RMSEA are between 0.062 and 0.076. When the groups are not assumed to be independent, the noninvariance conclusion error rates for ?RMSEA range from 0.000 to 0.015. Figure 8 provides a comparison of the noninvariance conclusion error rates for the decision rules when groups are assumed to be independent. With the exception of ?SRMR, all of the decision rules have noninvariance conclusion error rates under .05 regardless of the amount of correlation that may exist between the groups. Figure 9 shows that when groups are not assumed to be independent the noninvariance conclusion error rates of the various decision rules are all under 0.076 and these rates decrease as the correlation between groups increases. 25 Invariance Conclusion Error Rates Using the results from samples generated from the population models in which noninvariant indicators exist, the percentage of the ME/I tests that incorrectly identify these indicators as invariant were recorded. Incorrect conclusions of this type would result in a researcher believing that the factor loadings are equal across groups and would investigate further substantive hypotheses that may be untenable. When the groups are assumed to be independent, the invariance conclusion error rates for all decision rules are unaffected by the correlation between groups and decrease as ME/I test effect size (i.e., number of noninvariant indicators) increases for all decision rules (Figure 10). ?SRMR, , and ?RMSEA all have very low invariance conclusion error rates. When two or more of the five indicators were noninvariant, these three decision rules have invariance conclusion error rates under 0.10. ?CFI and Chen's Rule perform worse than any of the other decision rules. Even when all five indicators are noninvariant, the invariance conclusion error rates are over 0.20. When the independent groups assumption is removed, the number of noninvariant indicators still has a strong influence on the ME/I test results for all decision rules, but the invariance conclusion error rate also appears to have a quadratic relationship with between- groups correlation for all decision rules except for and ?RMSEA (Figures 11-15). In the case of (Figure 11), the uniqueness pair correlation appears to have no effect on the invariance conclusion error rates. For the ?CFI, ?SRMR, and Chen's decision rules (Figures 12, 13, and 15, respectively), invariance conclusion error rates reach a minimum value for all decision rules and effect sizes at a correlation of approximately 0.1. The invariance conclusion error rates for the ?RMSEA decision rule behave differently (Figure 14). With uniqueness pair 26 correlations of 0.3 or greater, the decision rule results in invariance conclusion error rates of 1.00 regardless of the number of noninvariant indicators. When comparing the invariance conclusion error rates of each decision rule at each level of group correlation (Figures 16-19), the test has the lowest invariance conclusion error rates across all levels of group correlation and number of noninvariant indicators. Parameter Estimate Bias To explore the effects of between groups dependence on the factor loading parameter estimates, a pair of 2 (Independent Groups Assumed vs Not Assumed) x 4 (Between Groups Factor Correlation: 0.0, 0.1, 0.3, and 0.5) x 4 (Corresponding Uniqueness Pair Correlation: 0.0, 0.1, 0.3, and 0.5) analyses of variance (ANOVA) of the factor loading parameter estimates were conducted. The first ANOVA (Table 9) evaluated the model for factor loadings in Group 2 that were invariant based on the population matrix. Although several effects are significant in the analysis, there is almost no practical significance (Overall Model R2 = 0.000088). The results were practically identical for the second ANOVA (Table 10) in which the noninvariant factor loadings were evaluated (Overall Model R2 = 0.000067). The sampling distributions of the factor loadings support these results as there does not appear to be any difference in factor loading parameter estimates other than the differences due to the lack of invariance. Figure 20 compares the sampling distributions of the factor loadings of an invariant indicator for Groups 1 and 2 when independent groups are assumed and not assumed. All four distributions are centered about the expected population factor loading of 0.80 and do not vary based on the independent groups assumption. Figure 21 presents the sampling distributions of the factor loadings of a noninvariant indicator for Groups 1 and 2 when independent groups are assumed and not assumed. The sampling distributions for the Group 1 27 factor loadings are centered about the expected population factor loading of 0.80 and the Group 2 factor loadings are centered about the expected population factor loading of 0.60. For both groups, the independent groups assumption appears to have no effect on the sampling distributions. Appendix D presents the complete set of descriptive statistics for the factor loadings. Parameter Estimate Standard Error Bias To explore the effects of between groups dependence on the factor loading parameter estimate standard errors, a pair of 2 (Independent Groups Assumed vs Not Assumed) x 4 (Between Groups Factor Correlation: 0.0, 0.1, 0.3, and 0.5) x 4 (Corresponding Uniqueness Pair Correlation: 0.0, 0.1, 0.3, and 0.5) analyses of variance (ANOVA) of the factor loading parameter estimate standard errors were conducted. The first ANOVA (Table 11) evaluated the model for the standard errors of factor loadings in Group 2 that were invariant based on the population matrix. Although several effects are significant in the analysis, there is almost no practical significance (Overall Model R2 = 0.000096). The results for the noninvariant factor loadings (Table 12) similarly showed no practical significance (Overall Model R2 = 0.004498). Appendix E presents the full set of descriptive statistics for the standard errors of the factor loadings. 28 Discussion Measurement invariance is an important factor to consider in any study which compares the responses of two groups on a particular measure. The most commonly used test of measurement invariance, the likelihood ratio test, assumes that the groups are independent. To this point, very little research has been done on the effect of a violation of the independent groups assumption. A Monte Carlo simulation was conducted to test Jones-Farmer's (2010) conclusion that such a violation would not affect the results of the likelihood ratio test. The simulation included several other ME/I decision rules based on CFI, SRMR, and RMSEA to serve as a replication of some of the findings from studies by Chen (2007) and Meade and Lautenschlager (2004) as well as an extension of the literature regarding the robustness of ME/I tests to independent groups assumption violations. Overall, the likelihood ratio test performed better than all other decision rules. It had the lowest error rates across most conditions. The results of the study supported the conclusion drawn by Jones-Farmer (2010). The likelihood ratio test was not affected by violations of the independent groups assumption. The results did not, however, support the conclusions of Chen (2007) at least within the context of the models evaluated by this simulation. The use of other decision rules did not add any predictive power to determining the existence of ME/I above that already provided by the likelihood ratio test. In fact, the use of various ME/I decision rules may lead to contradictory results that could mislead the researcher. The use of RMSEA was particularly problematic as it was affected most by the presence of among-groups dependence and the test assumptions. This study also addressed the concerns raised by Papadopoulos and Amemiya (2005) regarding the potential for among-groups dependence to affect parameter estimates and their 29 standard errors. Among-groups dependence was found to have no effect on the parameter estimates of the MCFA models in this study. There also appears to be no effect of among-groups dependence on the standard errors of those parameter estimates. These differences in results are most likely due to the nature of samples modeled by Papadopoulos and Amemiya. Their study was intended to model samples exhibiting a wide variety of distributional assumptions that do not necessarily coincide with the ones used in the current study. Limitations and Future Research There were several limitations to this research study. Some of these limitations concern the actual method of analyzing the structural equation models and ME/I tests. All analyses were run with ML estimation. It is important to know whether this robustness holds if other estimation methods are used. Furthermore, only the ME/I of factor loadings was tested. There are several other potential sources of ME/I and they should be tested similarly to ensure that the robustness to violations of the independent groups assumptions extends to them as well. Some decisions regarding the manipulation conditions and properties of the samples lead to limitations in the study. Firstly, a single sample size was used for all manipulation conditions and the group sample sizes were equal. A very common criticism of the likelihood ratio test is that it is too sensitive to sample size. Using a fairly large sample size of 500 in each group may have played to the strengths of the likelihood ratio test. Also, this large sample size may have influenced the results of the parameter estimate and standard error analyses. In future studies, it is recommended that various sample sizes and unequal group sample sizes be used to determine whether the conclusions of this study hold under a broader range of conditions. The population models were limited in scope as well. Only a one-factor model was used in all analyses. Future studies should have factors of various sizes. Furthermore, all 30 noninvariant indicators were lower in the second group. Meade and Lautenschlager (2004) found that the pattern of invariance affected the error rates of ME/I tests. If the noninvariant indicators were all uniformly lower in the second group, detecting the lack of invariance was more problematic than conditions in which the pattern of noninvariant indicators was mixed (i.e., some higher and some lower than in Group 1). In future robustness studies, this should be considered as a potential manipulation as well. Lastly, the difference between factor loadings of noninvariant indicators was set to 0.20 in all conditions. Using a variety of effect sizes should be considered in future studies to model an even broader range of studies in which researchers would use MCFA. Further research will be needed to generalize the results to a broader range of ME/I tests. Application to Experimental Research In studies similar to the ones modeled in this simulation, the likelihood ratio test was robust and was the best alternative for ME/I testing. Based on the results of this Monte Carlo simulation, researchers that wish to use another fit index are advised to supplement the conclusions with ?CFI. If there is any concern about whether the independent groups assumption is tenable for a particular sample, Jones-Farmer (2010) suggests a potential test for among-groups dependence using the noncentrality parameter. If among-groups dependence does exist, the conclusions drawn from ?SRMR and ?RMSEA would be unreliable and potentially misleading. Because of the problems with ?SRMR and ?RMSEA, this simulation does not support the use of the decision rule recommended by Chen (2007). Lastly, It is important to note that the results of this study relate only to the evaluation of ME/I. Although the likelihood ratio test was robust to independent groups assumption violations, the subsequent group comparisons most likely would be impacted by among-groups dependence. 31 Table 1 Summary of Population Covariance Matrix Manipulation Conditions Manipulation Value Note Test Assumptions: 0 Independent Groups Not Assumed 1 Independent Groups Assumed Number of Noninvariant Indicators: 0 100% Invariant 1 80% Invariant 2 60% Invariant 3 40% Invariant 4 20% Invariant 5 0% Invariant Correlation of Latent Factors 0.0 Uncorrelated 0.1 Low Correlation 0.3 Moderate Correlation 0.5 High Correlation Correlation of Uniqueness Pairs 0.0 Uncorrelated 0.1 Low Correlation 0.3 Moderate Correlation 0.5 High Correlation Table 2 Summary of Model-Implied Covariance Matrix Characteristics Used in Analyses Model Independent Groups Assumed? Factor Loadings Constrained? 1 Y N 2 Y Y 3 N N 4 N Y 32 Table 3 Main Effect of the Number of Noninvariant Indicators on Fit Index Values Fit Index F(5,959808) p ?2 ??2 221,846.00 < 0.05 53.34% ?CFI 231,120.00 < 0.05 53.94% ?SRMR 376,840.00 < 0.05 45.82% ?RMSEA 1,297,787.00 < 0.05 22.35% Table 3 Main Effect of the Latent Factor Correlation on Fit Index Values Fit Index F(3,959808) p ?2 ??2 162.33 < 0.05 0.02% ?CFI 11.18 < 0.05 0.00% ?SRMR 22.00 < 0.05 0.00% ?RMSEA 71,174.00 < 0.05 0.00% Table 5 Main Effect of the Uniqueness Pair Correlation on Fit Index Values Fit Index F(3,959808) P ?2 ??2 312.96 < 0.05 0.05% ?CFI 3,481.23 < 0.05 0.49% ?SRMR 14,482.40 < 0.05 1.06% ?RMSEA 168,057.00 < 0.05 5.68% 33 Table 6 Main Effect of the Test Assumptions on Fit Index Values Fit Index F(1,959808) P ?2 ??2 948.64 < 0.05 0.05% ?CFI 1,115.82 < 0.05 0.05% ?SRMR 1,047,829.00 < 0.05 25.48% ?RMSEA 1,297,787.00 < 0.05 34.52% Table 4 Interaction Effect of the Test Assumptions by the Number of Noninvariant Indicators on Fit Index Values Fit Index F(1,959808) P ?2 ??2 948.64 < 0.05 0.08% ?CFI 1,115.82 < 0.05 0.02% ?SRMR 1,047,829.00 < 0.05 3.09% ?RMSEA 1,297,787.00 < 0.05 5.96% Table 5 Interaction Effect of the Test Assumptions by the Uniqueness Pair Correlation on Fit Index Values Fit Index F(1,959808) P ?2 ??2 948.64 < 0.05 0.00% ?CFI 1,115.82 < 0.05 0.32% ?SRMR 1,047,829.00 < 0.05 0.92% ?RMSEA 1,297,787.00 < 0.05 5.02% 34 Table 9 ANOVA Results for Invariant Factor Loading Parameter Estimates Source DF SS MS F P Model 31 0.3540 0.0114 6.78 < 0.0001 Test Assumption (TA) 1 0.0947 0.0947 56.21 < 0.0001 Uniqueness r (Ur) 3 0.0779 0.0260 15.41 < 0.0001 Factor r (Fr) 3 0.0222 0.0074 4.39 0.0043 TA* Ur 3 0.0641 0.0214 12.67 < 0.0001 TA* Fr 3 0.0433 0.0144 8.56 < 0.0001 Ur* Fr 9 0.0266 0.0030 1.75 0.0717 TA* Ur* Fr 9 0.0254 0.0028 1.67 0.0893 Error 2,399,968 4043.0440 0.0017 Total 2,399,999 4043.3980 Table 10 ANOVA Results for Noninvariant Factor Loading Parameter Estimates Source DF SS MS F P Model 31 0.4733 0.0153 5.17 < 0.001 Test Assumption (TA) 1 0.0675 0.0675 22.87 < 0.001 Uniqueness r (Ur) 3 0.1139 0.0380 12.87 < 0.001 Factor r (Fr) 3 0.0009 0.0003 0.11 0.957 TA* Ur 3 0.1465 0.0488 16.55 <0.001 TA* Fr 3 0.0237 0.0079 2.68 0.045 Ur* Fr 9 0.0652 0.0072 2.46 0.009 TA* Ur* Fr 9 0.0555 0.0062 2.09 0.027 Error 2,399,968 7080.9705 0.0030 Total 2,399,999 7081.4438 35 Table 11 ANOVA Results for Invariant Factor Loading Parameter Estimate Standard Errors Source DF SS MS F P Model 31 0.0119 0.0004 7.42 < 0.0001 Test Assumption (TA) 1 0.0020 0.0020 37.95 < 0.0001 Uniqueness r (Ur) 3 0.0045 0.0015 28.66 < 0.0001 Factor r (Fr) 3 0.0003 0.0001 2.02 0.1082 TA* Ur 3 0.0004 0.0001 2.30 0.0754 TA* Fr 3 0.0032 0.0011 20.35 < 0.0001 Ur* Fr 9 0.0015 0.0002 3.31 0.0005 TA* Ur* Fr 9 0.0001 0.0000 0.25 0.9878 Error 2,399,968 124.7038 0.0001 Total 2,399,999 124.7158 Table 12 ANOVA Results for Noninvariant Factor Loading Parameter Estimate Standard Errors Source DF SS MS F P Model 31 0.2027 0.0065 349.80 < 0.0001 Test Assumption (TA) 1 0.0662 0.0662 3542.86 < 0.0001 Uniqueness r (Ur) 3 0.0031 0.0010 55.22 < 0.0001 Factor r (Fr) 3 0.0514 0.0171 916.61 < 0.0001 TA* Ur 3 0.0049 0.0016 86.51 < 0.0001 TA* Fr 3 0.0720 0.0240 1284.08 < 0.0001 Ur* Fr 9 0.0023 0.0003 13.82 < 0.0001 TA* Ur* Fr 9 0.0028 0.0003 16.60 < 0.0001 Error 2,399,968 44.8526 0.0000 Total 2,399,999 45.0552 36 x11 ?11 x12 ?12 x13 ?13 x14 ?14 x15 ?15 ?11 x21 ?21 x22 ?22 x23 ?23 x24 ?24 x25 ?25 ?21 Figure 1. One-factor model fit to two groups assuming independence between groups. x11 ?11 x12 ?12 x13 ?13 x14 ?14 x15 ?15 ?11 x21 ?21 x22 ?22 x23 ?23 x24 ?24 x25 ?25 ?21 Figure 2. One-factor model fit to two groups with a latent factor covariance between groups. 37 x11 ?11 x12 ?12 x13 ?13 x14 ?14 x15 ?15 ?11 x21 ?21 x22 ?22 x23 ?23 x24 ?24 x25 ?25 ?21 Figure 3. One-factor model fit to two groups with covariance between corresponding uniquenesses across groups. x11 ?11 x12 ?12 x13 ?13 x14 ?14 x15 ?15 ?11 x21 ?21 x22 ?22 x23 ?23 x24 ?24 x25 ?25 ?21 Figure 4. One-factor model fit to two groups with covariance between latent factors and corresponding uniquenesses across groups. 38 Figure 5. Comparison of the effect of test assumption on the noninvariance conclusion error rates for ??2 as a function of uniqueness pair correlation. Figure 6. Comparison of the effect of test assumption on the noninvariance conclusion error rates for ?SRMR as a function of uniqueness pair correlation. 39 Figure 7. Comparison of the effect of test assumption on the noninvariance conclusion error rates for ?RMSEA as a function of uniqueness pair correlation. Figure 8. Comparison of the noninvariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are assumed. 40 Figure 9. Comparison of the noninvariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are not assumed. Figure 10. Comparison of the invariance conclusion error rates for each decision rule as a function of uniqueness pair correlation when independent groups are assumed. 41 Figure 11. Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ??2 as a function of uniqueness pair correlation when independent groups are not assumed. Figure 12. Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?CFI as a function of uniqueness pair correlation when independent groups are not assumed. 42 Figure 13. Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?SRMR as a function of uniqueness pair correlation when independent groups are not assumed. Figure 14. Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for ?RMSEA as a function of uniqueness pair correlation when independent groups are not assumed. 43 Figure 15. Comparison of the effect of the number of noninvariant indicators on the invariance conclusion error rates for Chen's (2007) rule as a function of uniqueness pair correlation when independent groups are not assumed. Figure 16. Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have no correlation. 44 Figure 17. Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a low correlation. Figure 18. Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a moderate correlation. 45 Figure 19. Comparison of the invariance conclusion error rates for each decision rule as a function of the number of noninvariant items when independent groups are not assumed and uniqueness pairs have a high correlation. Figure 20. Comparison of the sampling distributions of invariant factor loadings. 46 Figure 21. Comparison of the sampling distributions of noninvariant factor loadings. 47 References Alwin, D. F., & Jackson, D. J. (1981). Applications of simultaneous factor analysis to issues of factorial invariance. In D. Jackson & E. Borgatta (Eds.), Factor analysis and measurement in sociological research: A multi-dimensional perspective (pp. 249-279). Beverly Hills, CA: Sage. Bagozzi, R. P., & Edwards, J. R. (1998). A general approach for representing constructs in organizational research. Organizational Research Methods, 1, 45-87. Bedeian, A. G., & Field, H. S. (2002). Assessing group change under conditions of anonymity and overlapping samples. Nursing Research, 51(1), 63-65. Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley & Sons. Butts, M. M., Vandenberg, R. J., & Williams, L. J. (2006, August). Investigating the susceptibility of measurement invariance tests: The effects of common method variance. Paper presented at the annual meeting of the Academy of Management, Atlanta, GA. Byrne, B. M. (1989). A primer of LISREL: Basic applications and programming for confirmatory factor analytic models. New York: Springer Verlag. Byrne, B. M. (1991). The Maslach Burnout Inventory: Validating factorial structure and invariance across intermediate, secondary, and university educators. Multivariate Behavioral Research, 26, 583-605. Byrne, B. M., & Baron. P. (1994). Measuring adolescent depression: Tests of equivalent factorial structure for English and French versions of the Beck Depression Inventory. Applied Psychology: An International Review, 43, 33-47. 48 Byrne, B. M., Baron, P., & Campbell, T. L. (1993). Measuring adolescent depression: Factorial validity and invariance of the Beck Depression Inventory across gender. Journal of Research on Adolescence, 3, 127-143. Byrne, B. M., Baron, P. & Campbell, T. L. (1994). The Beck Depression Inventory (French version): Testing for gender-invariant factorial structure for nonclinical adolescents. Journal of Adolescent Research, 9, 166-179. Byrne, B. M., Shavelson, R. J., & Muth?n, B. O. (1989). Testing for the equivalence of factor covariance and mean structures. Psychological Bulletin, 105, 456-466. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464-504. Cheung, G. W., & Rensvold, R. B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. Cronbach, L. J. (1992). Four Psychological Bulletin articles in perspective. Psychological Bulletin, 112, 389-392. Cronbach, L. J., & Furby, L. (1970). How should we measure ?change?: Or should we? Psychological Bulletin, 74, 68-80. Edwards, J. R. (2002). Alternatives to difference scores: Polynomial regression analysis and response surface methodology. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis. (pp. 350-400). San Francisco, CA: Jossey-Bass. 49 Finch, J. F., Okun, M. A., Barrera, M., Jr., Zautra, A. J., & Reich, J. W. (1989). Positive and negative social ties among older adults: Measurement models and the prediction of psychological distress and well-being. American Journal of Community Psychology, 17, 585-605. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491. French, B. F., & Finch, W. H. (2006). Confirmatory factor analytic procedures for the determination of measurement invariance. Structural Equation Modeling, 14(1), 611-635. Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental and Aging Research, 18, 117-144. J?reskog, K. G., & S?rbom, D. (1996). LISREL 8: User's reference guide. Chicago: Scientific Software International. Jones-Farmer, L.A. (2010). The effect of among group dependence on the invariance likelihood ratio test Structural Equation Modeling 17, 464-480. Little, T. D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32, 53-76. Lubke, G. H. & Muth?n, B. O. (2004). Applying multigroup confirmatory factor models for continuous outcomes to Likert scale data complicates meaningful group comparisons. Structural Equation Modeling, 11, 514-534. Marsh, H. W. (1994). Confirmatory factor analysis models of factorial invariance: A multifaceted approach. Structural Equation Modeling, 1, 5-34. 50 Marsh, H. W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept data: Between-group and within-group invariance constraints. Multivariate Behavioral Research, 28, 313-349. Mantzicopoulos, P. Y., French, B. F., & Maller, S. J. (2004). Factor structure of the pictorial scale of perceived competence and social acceptance with two pre-elementary samples. Child Development, 75, 1214-1228. McGaw, B., & J?reskog, K. G. (1971). Factorial invariance of ability measures in groups differing in intelligence and socio-economic status. British Journal of Mathematical and Statistical Psychology, 24, 154-168. Meade, A. W., & Bauer, D. J. (2007). power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling, 14(1), 611-635. Meade, A. W., & Lautenschlager, G. J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11, 60-72. Meredith, W. (1964). Notes on factorial invariance. Psychometrika, 29, 177-185. Messick, S. (1989). Meaning and values in test validation. The science and ethics of assessment. Education Researcher, 18, 5-11. Millsap, R. E., & Hartog, S. B. (1988). Alpha, beta, and gamma change in evaluation research: A structural equation approach. Journal of Applied Psychology, 73, 574-584. Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479-515. Mulaik, S.A. (1972). The foundations of factor analysis. New York: McGraw-Hill. Muth?n, L. K., & Muth?n, B. O. (2007). Mplus user?s guide, version 5. Los Angeles: Author. 51 Papadopoulos, S., & Amemiya, Y. (2005). Correlated samples with fixed and nonnormal latent variables. The Annals of Statistics, 33(6), 2732-2757. Raju, N. S., Laffitte, L. J., & Byrne, B. M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87(3), 517-529. Reise, S. P., Widaman, K. F., & Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552-566. Riordan, C. M., & Vandenberg, R. J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20(3), 643-671. Schmitt, N., Pulakos, E. D., & Lieblein, A. (1984). Comparison of three techniques to assess group-level beta and gamma change. Applied Psychological Measurement, 8, 249-260. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292-1306. Vandenberg, R.J. (2002). Toward a further understanding of and improvement in measurement invariance methods and practice. Organizational Research Methods, 5(2), 139-158. Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis of the measurement invariance literatures: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70. 52 Appendix A: Population Covariance Models 53 [ ] Figure A1. Population covariance matrix with zero noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. [ ] Figure A2. Population covariance matrix with zero noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. [ ] Figure A3. Population covariance matrix with zero noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. 54 [ ] Figure A4. Population covariance matrix with zero noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. [ ] Figure A5. Population covariance matrix with zero noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. [ ] Figure A6. Population covariance matrix with zero noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. 55 [ ] Figure A7. Population covariance matrix with zero noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. [ ] Figure A8. Population covariance matrix with zero noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. [ ] Figure A9. Population covariance matrix with zero noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. 56 [ ] Figure A10. Population covariance matrix with zero noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A11. Population covariance matrix with zero noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A12. Population covariance matrix with zero noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. 57 [ ] Figure A13. Population covariance matrix with zero noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. [ ] Figure A14. Population covariance matrix with zero noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. [ ] Figure A15. Population covariance matrix with zero noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. 58 [ ] Figure A16. Population covariance matrix with zero noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. [ ] Figure A17. Population covariance matrix with one noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. [ ] Figure A18. Population covariance matrix with one noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. 59 [ ] Figure A19. Population covariance matrix with one noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. [ ] Figure A20. Population covariance matrix with one noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. [ ] Figure A21. Population covariance matrix with one noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. 60 [ ] Figure A22. Population covariance matrix with one noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. [ ] Figure A23. Population covariance matrix with one noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. [ ] Figure A24. Population covariance matrix with one noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. 61 [ ] Figure A25. Population covariance matrix with one noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A26. Population covariance matrix with one noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A27. Population covariance matrix with one noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. 62 [ ] Figure A28. Population covariance matrix with one noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A29. Population covariance matrix with one noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. [ ] Figure A30. Population covariance matrix with one noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. 63 [ ] Figure A31. Population covariance matrix with one noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. [ ] Figure A32. Population covariance matrix with one noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. [ ] Figure A33. Population covariance matrix with two noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. 64 [ ] Figure A34. Population covariance matrix with two noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. [ ] Figure A35. Population covariance matrix with two noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. [ ] Figure A36. Population covariance matrix with two noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. 65 [ ] Figure A37. Population covariance matrix with two noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. [ ] Figure A38. Population covariance matrix with two noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. [ ] Figure A39. Population covariance matrix with two noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. 66 [ ] Figure A40. Population covariance matrix with two noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. [ ] Figure A41. Population covariance matrix with two noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A42. Population covariance matrix with two noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. 67 [ ] Figure A43. Population covariance matrix with two noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A44. Population covariance matrix with two noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A45. Population covariance matrix with two noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. 68 [ ] Figure A46. Population covariance matrix with two noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. [ ] Figure A47. Population covariance matrix with two noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. [ ] Figure A48. Population covariance matrix with two noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. 69 [ ] Figure A49. Population covariance matrix with three noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. [ ] Figure A50. Population covariance matrix with three noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. [ ] Figure A51. Population covariance matrix with three noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. 70 [ ] Figure A52. Population covariance matrix with three noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. [ ] Figure A53. Population covariance matrix with three noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. [ ] Figure A54. Population covariance matrix with three noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. 71 [ ] Figure A55. Population covariance matrix with three noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. [ ] Figure A56. Population covariance matrix with three noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. [ ] Figure A57. Population covariance matrix with three noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. 72 [ ] Figure A58. Population covariance matrix with three noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A59. Population covariance matrix with three noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A60. Population covariance matrix with three noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. 73 [ ] Figure A61. Population covariance matrix with three noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. [ ] Figure A62. Population covariance matrix with three noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. [ ] Figure A63. Population covariance matrix with three noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. 74 [ ] Figure A64. Population covariance matrix with three noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. [ ] Figure A65. Population covariance matrix with four noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. [ ] Figure A66. Population covariance matrix with four noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. 75 [ ] Figure A67. Population covariance matrix with four noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. [ ] Figure A68. Population covariance matrix with four noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. [ ] Figure A69. Population covariance matrix with four noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. 76 [ ] Figure A70. Population covariance matrix with four noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. [ ] Figure A71. Population covariance matrix with four noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. [ ] Figure A72. Population covariance matrix with four noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. 77 [ ] Figure A73. Population covariance matrix with four noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A74. Population covariance matrix with four noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A75. Population covariance matrix with four noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. 78 [ ] Figure A76. Population covariance matrix with four noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A77. Population covariance matrix with four noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. [ ] Figure A78. Population covariance matrix with four noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. 79 [ ] Figure A79. Population covariance matrix with four noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. [ ] Figure A80. Population covariance matrix with four noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. [ ] Figure A81. Population covariance matrix with five noninvariant indicators, no latent factor correlation, and no uniqueness pair correlation. 80 [ ] Figure A82. Population covariance matrix with five noninvariant indicators, low latent factor correlation, and no uniqueness pair correlation. [ ] Figure A83. Population covariance matrix with five noninvariant indicators, moderate latent factor correlation, and no uniqueness pair correlation. [ ] Figure A84. Population covariance matrix with five noninvariant indicators, high latent factor correlation, and no uniqueness pair correlation. 81 [ ] Figure A85. Population covariance matrix with five noninvariant indicators, no latent factor correlation, and low uniqueness pair correlation. [ ] Figure A86. Population covariance matrix with five noninvariant indicators, low latent factor correlation, and low uniqueness pair correlation. [ ] Figure A87. Population covariance matrix with five noninvariant indicators, moderate latent factor correlation, and low uniqueness pair correlation. 82 [ ] Figure A88. Population covariance matrix with five noninvariant indicators, high latent factor correlation, and low uniqueness pair correlation. [ ] Figure A89. Population covariance matrix with five noninvariant indicators, no latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A90. Population covariance matrix with five noninvariant indicators, low latent factor correlation, and moderate uniqueness pair correlation. 83 [ ] Figure A91. Population covariance matrix with five noninvariant indicators, moderate latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A92. Population covariance matrix with five noninvariant indicators, high latent factor correlation, and moderate uniqueness pair correlation. [ ] Figure A93. Population covariance matrix with five noninvariant indicators, no latent factor correlation, and high uniqueness pair correlation. 84 [ ] Figure A94. Population covariance matrix with five noninvariant indicators, low latent factor correlation, and high uniqueness pair correlation. [ ] Figure A95. Population covariance matrix with five noninvariant indicators, moderate latent factor correlation, and high uniqueness pair correlation. [ ] Figure A96. Population covariance matrix with five noninvariant indicators, high latent factor correlation, and high uniqueness pair correlation. 85 Appendix B: ANOVA of the Factors Affecting ME/I Test Statistic/Indices 86 Table B1 ANOVA of the Factors Affecting ??2 Source df SS MS F eta Model 191 84804889 444005 5863.52 53.85% TA 1 71834 71834 948.64 0.05% NI 5 83994540 16798908 221846.00 53.34% Ur 3 71095 23698 312.96 0.05% Fr 3 36877 12292 162.33 0.02% TA * NI 5 132760 26552 350.65 0.08% TA * Ur 3 2527 842 11.12 0.00% TA * Fr 3 123986 41329 545.79 0.08% NI * Ur 15 14439 963 12.71 0.01% NI * Fr 15 162148 10810 142.75 0.10% Ur * Fr 9 14966 1663 21.96 0.01% TA * NI * Ur 15 12842 856 11.31 0.01% TA * NI * Fr 15 150272 10018 132.30 0.10% TA * Ur * Fr 9 253 28 0.37 0.00% NI * Ur * Fr 45 8845 197 2.60 0.01% TA * NI * Ur * Fr 45 7505 167 2.20 0.00% Error 959808 72679742 76 Corrected Total 959999 157484631 Note: TA ? Test assumptions (Independent Groups Assumed vs. Not Assumed); NI ? Number of Noninvariant Indicators (0, 1, 2, 3, 4, 5); Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 87 Table B2 ANOVA of the Factors Affecting ?CFI Source df SS MS F eta Model 191 31.91613 0.167100 6190.49 55.20% TA 1 0.03012 0.030119 1115.82 0.05% NI 5 31.19319 6.238639 231120 53.94% Ur 3 0.28191 0.093969 3481.23 0.49% Fr 3 0.00091 0.000302 11.18 0.00% TA * NI 5 0.01337 0.002674 99.06 0.02% TA * Ur 3 0.18726 0.062421 2312.5 0.32% TA * Fr 3 0.01007 0.003356 124.31 0.02% NI * Ur 15 0.06901 0.004600 170.43 0.12% NI * Fr 15 0.03102 0.002068 76.61 0.05% Ur * Fr 9 0.00541 0.000601 22.26 0.01% TA * NI * Ur 15 0.05991 0.003994 147.96 0.10% TA * NI * Fr 15 0.03205 0.002137 79.15 0.06% TA * Ur * Fr 9 0.00030 0.000033 1.23 0.00% NI * Ur * Fr 45 0.00088 0.000020 0.73 0.00% TA * NI * Ur * Fr 45 0.00074 0.000016 0.61 0.00% Error 959808 25.90814 0.000027 Corrected Total 959999 57.82426 Note: TA ? Test assumptions (Independent Groups Assumed vs. Not Assumed); NI ? Number of Noninvariant Indicators (0, 1, 2, 3, 4, 5); Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 88 Table B3 ANOVA of the Factors Affecting ?SRMR Source df SS MS F eta Model 191 866.526 4.53679 16504.80 76.66% TA 1 288.024 288.02441 1047829.00 25.48% NI 5 517.924 103.58477 376840.00 45.82% Ur 3 11.943 3.98089 14482.40 1.06% Fr 3 0.018 0.00605 22.00 0.00% TA * NI 5 34.984 6.99685 25454.50 3.09% TA * Ur 3 10.356 3.45193 12558.10 0.92% TA * Fr 3 0.227 0.07563 275.13 0.02% NI * Ur 15 1.245 0.08302 302.03 0.11% NI * Fr 15 0.120 0.00801 29.15 0.01% Ur * Fr 9 0.054 0.00598 21.76 0.00% TA * NI * Ur 15 1.577 0.10513 382.47 0.14% TA * NI * Fr 15 0.017 0.00115 4.18 0.00% TA * Ur * Fr 9 0.010 0.00108 3.92 0.00% NI * Ur * Fr 45 0.006 0.00012 0.45 0.00% TA * NI * Ur * Fr 45 0.022 0.00048 1.74 0.00% Error 959808 263.829 0.00028 Corrected Total 959999 1130.356 Note: TA ? Test assumptions (Independent Groups Assumed vs. Not Assumed); NI ? Number of Noninvariant Indicators (0, 1, 2, 3, 4, 5); Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 89 Table B4 ANOVA of the Factors Affecting ?RMSEA Source df SS MS F eta Model 191 382.8377 2.91698 15474.67 74.47% TA 1 177.4601 2.00439 14658.30 34.52% NI 5 114.9008 177.46012 1297787.00 22.35% Ur 3 29.1971 22.98016 168057.00 5.68% Fr 3 0.0019 9.73237 71174.00 0.00% TA * NI 5 30.6240 0.00064 4.71 5.96% TA * Ur 3 25.8311 6.12480 44791.40 5.02% TA * Fr 3 0.0520 8.61038 62968.70 0.01% NI * Ur 15 2.1582 0.01733 126.73 0.42% NI * Fr 15 0.0418 0.14388 1052.23 0.01% Ur * Fr 9 0.0266 0.00279 20.38 0.01% TA * NI * Ur 15 2.5199 0.00296 21.63 0.49% TA * NI * Fr 15 0.0128 0.16799 1228.54 0.00% TA * Ur * Fr 9 0.0010 0.00085 6.22 0.00% NI * Ur * Fr 45 0.0018 0.00011 0.83 0.00% TA * NI * Ur * Fr 45 0.0085 0.00004 0.29 0.00% Error 959808 131.2447 0.00019 1.38 Corrected Total 959999 514.0824 0.00014 Note: TA ? Test assumptions (Independent Groups Assumed vs. Not Assumed); NI ? Number of Noninvariant Indicators (0, 1, 2, 3, 4, 5); Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 90 Appendix C: Error Rate Tables 91 Table C1 Invariance Conclusion Error Rates for All Decision Rules by Group Correlation ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 4.76% 4.70% 0.00% 0.00% 3.66% 27.10% 1.50% 7.56% 0.00% 0.00% 0.1 4.78% 4.68% 0.00% 0.00% 3.50% 26.70% 1.40% 7.44% 0.00% 0.00% 0.3 4.82% 4.52% 0.00% 0.00% 3.16% 25.30% 1.48% 7.24% 0.00% 0.00% 0.5 5.14% 4.06% 0.00% 0.00% 2.52% 22.70% 1.32% 7.06% 0.00% 0.00% 0.1 0.0 4.70% 4.68% 0.00% 0.00% 2.96% 27.00% 0.28% 7.40% 0.00% 0.00% 0.1 4.50% 4.48% 0.00% 0.00% 2.66% 26.50% 0.30% 7.20% 0.00% 0.00% 0.3 4.24% 4.18% 0.00% 0.00% 2.28% 24.70% 0.20% 6.78% 0.00% 0.00% 0.5 3.80% 3.52% 0.00% 0.00% 1.78% 21.70% 0.16% 6.00% 0.00% 0.00% 0.3 0.0 4.32% 4.38% 0.00% 0.00% 0.90% 26.60% 0.00% 6.78% 0.00% 0.00% 0.1 3.80% 4.00% 0.00% 0.00% 0.90% 25.90% 0.00% 6.18% 0.00% 0.00% 0.3 2.84% 3.08% 0.00% 0.00% 0.64% 23.20% 0.00% 5.02% 0.00% 0.00% 0.5 2.06% 2.34% 0.00% 0.00% 0.38% 19.80% 0.00% 4.22% 0.00% 0.00% 0.5 0.0 3.68% 3.86% 0.00% 0.00% 0.18% 25.80% 0.00% 6.22% 0.00% 0.00% 0.1 2.96% 3.18% 0.00% 0.00% 0.16% 24.80% 0.00% 5.20% 0.00% 0.00% 0.3 1.78% 2.30% 0.00% 0.00% 0.12% 22.10% 0.00% 3.80% 0.00% 0.00% 0.5 0.86% 1.08% 0.00% 0.00% 0.02% 17.50% 0.00% 2.16% 0.00% 0.00% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 92 Table C2 Noninvariance Conclusion Error Rates for All Decision Rules by Group Correlation When One Indicator Is Invariant ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 25.10% 25.10% 89.90% 88.80% 75.00% 16.70% 58.80% 30.10% 90.28% 88.80% 0.1 24.90% 25.00% 89.80% 89.00% 74.70% 16.60% 58.60% 30.10% 90.18% 89.00% 0.3 23.90% 24.90% 90.24% 88.90% 71.70% 16.60% 58.20% 30.10% 90.54% 88.90% 0.5 24.90% 24.90% 89.90% 88.90% 75.10% 16.50% 59.00% 30.10% 90.30% 88.90% 0.1 0.0 25.00% 25.00% 86.40% 89.00% 79.10% 16.60% 89.20% 29.70% 90.26% 89.00% 0.1 25.30% 25.10% 86.80% 89.10% 78.90% 16.40% 89.40% 30.00% 90.32% 89.10% 0.3 24.60% 24.90% 87.70% 89.30% 76.40% 16.60% 89.50% 30.00% 90.00% 89.30% 0.5 24.20% 25.30% 89.20% 89.90% 71.10% 16.90% 89.30% 30.90% 90.04% 89.90% 0.3 0.0 25.30% 25.20% 90.28% 89.50% 91.86% 16.00% 100.00% 30.10% 96.26% 89.50% 0.1 26.00% 25.50% 91.06% 89.90% 92.02% 16.00% 100.00% 30.50% 96.56% 89.90% 0.3 26.20% 25.70% 92.62% 90.48% 91.98% 16.30% 100.00% 31.00% 96.78% 90.48% 0.5 26.40% 26.00% 94.46% 91.18% 91.32% 17.10% 100.00% 32.10% 96.60% 91.18% 0.5 0.0 25.50% 25.00% 95.50% 90.12% 97.40% 16.20% 100.00% 30.50% 99.02% 90.12% 0.1 26.50% 25.30% 96.20% 90.54% 97.62% 16.10% 100.00% 30.90% 99.18% 90.54% 0.3 28.00% 26.10% 97.82% 91.84% 98.04% 16.80% 100.00% 31.60% 99.48% 91.84% 0.5 29.40% 26.90% 98.84% 93.26% 98.44% 17.50% 100.00% 33.20% 99.58% 93.26% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 93 Table C3 Noninvariance Conclusion Error Rates for All Decision Rules by Group Correlation When Two Indicators Are Invariant ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 5.40% 5.40% 57.60% 53.60% 39.50% 3.20% 28.40% 9.90% 58.80% 53.60% 0.1 5.40% 5.40% 57.40% 53.60% 39.00% 3.30% 28.60% 9.60% 58.60% 53.60% 0.3 5.10% 5.30% 58.30% 53.60% 34.40% 3.30% 27.70% 9.70% 59.30% 53.60% 0.5 4.50% 5.30% 58.00% 54.30% 25.90% 2.90% 26.50% 9.60% 58.40% 54.30% 0.1 0.0 5.30% 5.30% 48.40% 53.50% 44.00% 3.40% 70.20% 9.70% 56.20% 53.50% 0.1 5.20% 5.20% 48.40% 53.50% 44.20% 3.30% 70.10% 9.50% 56.10% 53.50% 0.3 5.20% 5.10% 49.10% 54.20% 39.90% 3.10% 69.60% 9.50% 55.00% 54.20% 0.5 4.40% 5.10% 50.30% 54.70% 31.80% 2.70% 68.50% 9.30% 53.10% 54.70% 0.3 0.0 5.30% 5.20% 54.60% 53.70% 67.20% 3.30% 100.00% 9.20% 73.90% 53.70% 0.1 5.30% 5.20% 55.60% 54.00% 67.50% 3.20% 100.00% 9.10% 74.30% 54.00% 0.3 4.80% 4.90% 58.50% 55.00% 66.10% 3.20% 100.00% 8.70% 74.00% 55.00% 0.5 4.40% 4.80% 61.50% 56.20% 61.60% 2.60% 100.00% 8.40% 71.70% 56.20% 0.5 0.0 5.40% 5.10% 69.80% 53.70% 86.00% 3.30% 100.00% 9.10% 89.70% 53.70% 0.1 5.30% 5.00% 72.10% 54.50% 86.50% 3.20% 100.00% 8.80% 90.58% 54.50% 0.3 5.00% 4.80% 76.70% 56.10% 87.00% 3.00% 100.00% 8.70% 91.10% 56.10% 0.5 4.60% 4.50% 81.10% 58.20% 86.50% 2.40% 100.00% 8.80% 90.64% 58.20% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 94 Table C4 Noninvariance Conclusion Error Rates for All Decision Rules by Group Correlation When Three Indicators Are Invariant ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 1.70% 1.70% 35.90% 30.10% 16.10% 0.80% 17.50% 5.00% 36.90% 30.10% 0.1 1.90% 1.80% 35.70% 30.00% 15.40% 0.70% 17.20% 4.90% 36.70% 30.00% 0.3 1.60% 1.80% 34.50% 29.60% 12.80% 0.50% 16.50% 4.80% 35.20% 29.60% 0.5 1.20% 1.60% 32.30% 29.60% 8.30% 0.40% 14.30% 4.80% 32.70% 29.60% 0.1 0.0 1.80% 1.80% 25.30% 30.20% 18.80% 0.70% 55.30% 5.00% 29.90% 30.20% 0.1 1.70% 1.80% 25.10% 30.10% 18.30% 0.70% 55.00% 5.10% 29.50% 30.10% 0.3 1.60% 1.60% 24.60% 30.10% 15.80% 0.60% 53.70% 4.90% 27.70% 30.10% 0.5 1.20% 1.40% 24.00% 29.80% 10.70% 0.40% 50.50% 4.90% 25.40% 29.80% 0.3 0.0 1.80% 1.80% 30.00% 30.20% 38.00% 0.70% 99.98% 4.70% 44.80% 30.20% 0.1 1.70% 1.70% 30.50% 30.40% 37.90% 0.70% 99.98% 4.70% 44.80% 30.40% 0.3 1.40% 1.60% 31.50% 30.40% 35.80% 0.50% 99.98% 4.50% 43.60% 30.40% 0.5 1.10% 1.40% 31.10% 30.30% 30.10% 0.30% 99.98% 4.30% 38.50% 30.30% 0.5 0.0 1.90% 1.80% 44.60% 30.90% 62.40% 0.60% 100.00% 4.60% 67.60% 30.90% 0.1 1.60% 1.70% 45.80% 30.80% 62.80% 0.50% 100.00% 4.60% 68.00% 30.80% 0.3 1.50% 1.60% 48.20% 31.00% 62.20% 0.40% 100.00% 4.50% 67.30% 31.00% 0.5 1.00% 1.20% 49.00% 31.00% 59.20% 0.30% 100.00% 4.10% 64.00% 31.00% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 95 Table C5 Noninvariance Conclusion Error Rates for All Decision Rules by Group Correlation When Four Indicators Are Invariant ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 1.70% 1.70% 29.20% 23.90% 5.00% 0.10% 16.40% 4.50% 29.40% 23.90% 0.1 1.60% 1.70% 28.70% 23.80% 4.90% 0.10% 16.40% 4.60% 28.90% 23.80% 0.3 1.30% 1.60% 26.50% 23.50% 3.90% 0.10% 14.80% 4.40% 26.60% 23.50% 0.5 0.70% 1.40% 22.30% 23.00% 2.10% 0.10% 11.10% 3.90% 22.40% 23.00% 0.1 0.0 1.70% 1.70% 19.30% 23.80% 6.10% 0.10% 52.10% 4.50% 20.10% 23.80% 0.1 1.60% 1.60% 19.10% 23.80% 6.20% 0.10% 51.60% 4.40% 19.80% 23.80% 0.3 1.30% 1.50% 17.50% 23.40% 5.00% 0.10% 48.70% 4.20% 17.80% 23.40% 0.5 0.70% 1.30% 15.00% 23.10% 3.20% 0.10% 42.40% 4.00% 15.20% 23.10% 0.3 0.0 1.60% 1.50% 23.20% 23.70% 17.10% 0.10% 99.98% 4.20% 27.30% 23.70% 0.1 1.50% 1.50% 23.20% 23.70% 16.70% 0.10% 99.98% 4.10% 27.00% 23.70% 0.3 1.10% 1.40% 22.10% 23.30% 14.80% 0.10% 99.98% 4.10% 25.00% 23.30% 0.5 0.60% 1.10% 18.40% 23.00% 11.30% 0.10% 99.98% 3.70% 20.00% 23.00% 0.5 0.0 1.50% 1.50% 35.80% 23.90% 35.80% 0.10% 100.00% 4.00% 44.60% 23.90% 0.1 1.40% 1.40% 36.30% 24.20% 35.90% 0.10% 100.00% 3.80% 44.80% 24.20% 0.3 1.00% 1.20% 35.50% 24.00% 34.50% 0.10% 100.00% 4.00% 42.40% 24.00% 0.5 0.50% 0.90% 30.60% 23.70% 30.70% 0.00% 100.00% 3.70% 36.40% 23.70% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 96 Table C5 Noninvariance Conclusion Error Rates for All Decision Rules by Group Correlation When Five Indicators Are Invariant ??2 ?CFI ?SRMR ?RMSEA Chen Ur Fr N I N I N I N I N I 0.0 0.0 1.30% 1.40% 25.00% 20.20% 0.90% 0.00% 16.40% 4.40% 25.00% 20.20% 0.1 1.30% 1.30% 24.40% 20.40% 0.90% 0.00% 16.20% 4.40% 24.40% 20.40% 0.3 0.90% 1.20% 22.10% 19.60% 0.70% 0.00% 13.90% 4.50% 22.10% 19.60% 0.5 0.40% 0.90% 17.30% 17.90% 0.50% 0.00% 10.10% 3.90% 17.30% 17.90% 0.1 0.0 1.30% 1.40% 15.30% 20.20% 1.40% 0.00% 50.40% 4.40% 15.30% 20.20% 0.1 1.30% 1.30% 15.20% 20.20% 1.30% 0.00% 49.80% 4.50% 15.20% 20.20% 0.3 0.90% 1.20% 13.10% 19.90% 1.10% 0.00% 45.70% 4.30% 13.10% 19.90% 0.5 0.40% 0.90% 9.30% 18.20% 0.70% 0.00% 37.50% 4.00% 9.30% 18.20% 0.3 0.0 1.30% 1.30% 19.00% 20.30% 5.00% 0.00% 99.98% 4.10% 19.10% 20.30% 0.1 1.30% 1.40% 18.90% 20.40% 4.90% 0.00% 99.98% 4.20% 19.00% 20.40% 0.3 0.90% 1.20% 16.20% 20.10% 4.10% 0.00% 99.98% 4.20% 16.20% 20.10% 0.5 0.40% 0.90% 11.00% 19.00% 3.10% 0.00% 99.98% 3.90% 11.00% 19.00% 0.5 0.0 1.50% 1.60% 30.00% 20.50% 15.40% 0.00% 100.00% 4.20% 30.40% 20.50% 0.1 1.30% 1.50% 29.60% 21.00% 15.20% 0.00% 100.00% 4.10% 29.90% 21.00% 0.3 0.80% 1.20% 25.80% 20.20% 14.00% 0.00% 100.00% 3.90% 26.10% 20.20% 0.5 0.40% 0.90% 17.50% 19.70% 11.80% 0.00% 100.00% 3.70% 18.10% 19.70% Note: I ? Independent Groups Assumed; N ? Independent Groups Not Assumed; Ur ? Uniqueness Pair Correlation (0.0, 0.1, 0.3, 0.5); Fr ? Latent Factor Correlation (0.0, 0.1, 0.3, 0.5); 97 Appendix D: Descriptive Statistics of the Sample Parameter Estimates 98 Table D1 Descriptive Statistics of the Invariant Factor Loadings Independent Groups Not Assumed Independent Groups Assumed Ur Fr M (SD) 99% CI M (SD) 99% CI 0.0 0.0 0.798284 (0.001683) (0.798268, 0.798299) 0.798298 (0.001683) (0.798282, 0.798313) 0.1 0.798313 (0.001687) (0.798297, 0.798329) 0.798318 (0.001687) (0.798302, 0.798334) 0.3 0.798320 (0.001695) (0.798304, 0.798336) 0.798366 (0.001702) (0.798350, 0.798382) 0.5 0.798309 (0.001692) (0.798293, 0.798325) 0.798374 (0.001713) (0.798358, 0.798391) 0.1 0.0 0.798263 (0.001683) (0.798247, 0.798278) 0.798302 (0.001682) (0.798286, 0.798318) 0.1 0.798221 (0.001687) (0.798205, 0.798237) 0.798292 (0.001687) (0.798276, 0.798308) 0.3 0.798153 (0.001695) (0.798137, 0.798169) 0.798357 (0.001704) (0.798341, 0.798373) 0.5 0.798019 (0.001701) (0.798003, 0.798035) 0.798419 (0.001725) (0.798403, 0.798435) 0.3 0.0 0.798184 (0.001680) (0.798168, 0.798200) 0.798287 (0.001679) (0.798272, 0.798303) 0.1 0.798029 (0.001686) (0.798013, 0.798045) 0.798292 (0.001686) (0.798276, 0.798308) 0.3 0.797717 (0.001703) (0.797701, 0.797733) 0.798315 (0.001706) (0.798299, 0.798331) 0.5 0.797385 (0.001715) (0.797369, 0.797401) 0.798397 (0.001732) (0.798381, 0.798413) 0.5 0.0 0.798027 (0.001683) (0.798011, 0.798043) 0.798279 (0.001680) (0.798264, 0.798295) 0.1 0.797772 (0.001693) (0.797756, 0.797788) 0.798273 (0.001688) (0.798257, 0.798289) 0.3 0.797209 (0.001721) (0.797193, 0.797226) 0.798282 (0.001711) (0.798265, 0.798298) 0.5 0.796651 (0.001753) (0.796635, 0.796668) 0.798361 (0.001744) (0.798344, 0.798377) Note: Ur ? Uniqueness Pair Correlation; Fr ? Latent Factor Correlation; n = 75000; 99 Table D2 Descriptive Statistics of the Noninvariant Factor Loadings Independent Groups Not Assumed Independent Groups Assumed Ur Fr M (SD) 99% CI M (SD) 99% CI 0.0 0.0 0.600947 (0.002982) (0.600919, 0.600975) 0.600999 (0.002981) (0.600971, 0.601027) 0.1 0.600911 (0.002983) (0.600883, 0.600939) 0.600971 (0.002983) (0.600943, 0.600999) 0.3 0.600588 (0.002926) (0.600560, 0.600615) 0.600867 (0.002978) (0.600839, 0.600895) 0.5 0.600347 (0.002837) (0.600321, 0.600374) 0.600765 (0.002969) (0.600737, 0.600793) 0.1 0.0 0.600931 (0.002981) (0.600903, 0.600959) 0.600973 (0.002981) (0.600945, 0.601001) 0.1 0.600941 (0.002976) (0.600913, 0.600969) 0.600919 (0.002980) (0.600891, 0.600948) 0.3 0.600864 (0.002926) (0.600836, 0.600891) 0.600813 (0.002974) (0.600785, 0.600841) 0.5 0.600664 (0.002812) (0.600638, 0.600691) 0.600709 (0.002963) (0.600681, 0.600737) 0.3 0.0 0.601004 (0.002987) (0.600976, 0.601032) 0.600911 (0.002983) (0.600883, 0.600939) 0.1 0.601206 (0.002983) (0.601178, 0.601234) 0.600916 (0.002988) (0.600888, 0.600944) 0.3 0.601382 (0.002923) (0.601355, 0.601410) 0.600814 (0.002979) (0.600786, 0.600842) 0.5 0.601632 (0.002823) (0.601605, 0.601658) 0.600718 (0.002970) (0.600690, 0.600746) 0.5 0.0 0.601223 (0.002999) (0.601194, 0.601251) 0.600892 (0.002991) (0.600864, 0.600920) 0.1 0.601527 (0.002991) (0.601499, 0.601555) 0.600840 (0.002989) (0.600812, 0.600868) 0.3 0.602095 (0.002946) (0.602067, 0.602122) 0.600803 (0.002990) (0.600774, 0.600831) 0.5 0.602724 (0.002862) (0.602697, 0.602751) 0.600712 (0.002973) (0.600684, 0.600740) Note: Ur ? Uniqueness Pair Correlation; Fr ? Latent Factor Correlation; n = 75000; 100 Appendix E: Descriptive Statistics of the Parameter Estimate Standard Errors 101 Table E1 Descriptive Statistics of the Standard Errors of the Invariant Factor Loadings Independent Groups Not Assumed Independent Groups Assumed Ur Fr M (SD) 99% CI M (SD) 99% CI 0.0 0.0 0.037564 (0.000046) (0.037563, 0.037564) 0.037556 (0.000046) (0.037556, 0.037557) 0.1 0.037585 (0.000046) (0.037585, 0.037586) 0.037559 (0.000046) (0.037558, 0.037559) 0.3 0.037755 (0.000042) (0.037754, 0.037755) 0.037557 (0.000046) (0.037557, 0.037558) 0.5 0.037950 (0.000037) (0.037949, 0.037950) 0.037534 (0.000046) (0.037534, 0.037535) 0.1 0.0 0.037566 (0.000046) (0.037566, 0.037567) 0.037563 (0.000046) (0.037562, 0.037563) 0.1 0.037570 (0.000046) (0.037570, 0.037570) 0.037551 (0.000046) (0.037550, 0.037551) 0.3 0.037714 (0.000042) (0.037714, 0.037715) 0.037546 (0.000046) (0.037545, 0.037546) 0.5 0.037951 (0.000035) (0.037951, 0.037951) 0.037498 (0.000046) (0.037498, 0.037498) 0.3 0.0 0.037544 (0.000046) (0.037543, 0.037544) 0.037544 (0.000046) (0.037544, 0.037545) 0.1 0.037526 (0.000045) (0.037525, 0.037526) 0.037515 (0.000046) (0.037515, 0.037515) 0.3 0.037633 (0.000041) (0.037633, 0.037634) 0.037463 (0.000046) (0.037463, 0.037463) 0.5 0.037846 (0.000035) (0.037846, 0.037847) 0.037419 (0.000047) (0.037418, 0.037419) 0.5 0.0 0.037511 (0.000046) (0.037510, 0.037511) 0.037530 (0.000046) (0.037530, 0.037531) 0.1 0.037485 (0.000045) (0.037484, 0.037485) 0.037484 (0.000046) (0.037484, 0.037485) 0.3 0.037557 (0.000041) (0.037556, 0.037557) 0.037415 (0.000047) (0.037415, 0.037416) 0.5 0.037749 (0.000034) (0.037748, 0.037749) 0.037331 (0.000047) (0.037330, 0.037331) Note: Ur ? Uniqueness Pair Correlation; Fr ? Latent Factor Correlation; n = 75000; 102 Table E2 Descriptive Statistics of the Standard Errors of the Noninvariant Factor Loadings Independent Groups Not Assumed Independent Groups Assumed Ur Fr M (SD) 99% CI M (SD) 99% CI 0.0 0.0 0.050921 (0.000020) (0.050921, 0.050921) 0.050927 (0.000020) (0.050927, 0.050927) 0.1 0.050890 (0.000019) (0.050890, 0.050890) 0.050928 (0.000020) (0.050928, 0.050928) 0.3 0.050686 (0.000017) (0.050686, 0.050686) 0.050950 (0.000019) (0.050950, 0.050950) 0.5 0.050172 (0.000015) (0.050171, 0.050172) 0.050994 (0.000019) (0.050994, 0.050994) 0.1 0.0 0.050921 (0.000020) (0.050920, 0.050921) 0.050929 (0.000020) (0.050928, 0.050929) 0.1 0.050885 (0.000019) (0.050885, 0.050886) 0.050938 (0.000019) (0.050938, 0.050939) 0.3 0.050636 (0.000017) (0.050636, 0.050636) 0.050963 (0.000019) (0.050963, 0.050963) 0.5 0.050090 (0.000014) (0.050090, 0.050090) 0.051006 (0.000019) (0.051006, 0.051006) 0.3 0.0 0.050914 (0.000020) (0.050914, 0.050914) 0.050945 (0.000020) (0.050945, 0.050945) 0.1 0.050842 (0.000019) (0.050842, 0.050842) 0.050939 (0.000020) (0.050939, 0.050940) 0.3 0.050538 (0.000017) (0.050537, 0.050538) 0.050968 (0.000019) (0.050968, 0.050968) 0.5 0.049883 (0.000014) (0.049883, 0.049884) 0.051012 (0.000019) (0.051012, 0.051012) 0.5 0.0 0.050879 (0.000020) (0.050879, 0.050879) 0.050932 (0.000020) (0.050932, 0.050932) 0.1 0.050790 (0.000019) (0.050790, 0.050790) 0.050949 (0.000019) (0.050949, 0.050950) 0.3 0.050409 (0.000017) (0.050408, 0.050409) 0.050981 (0.000019) (0.050981, 0.050981) 0.5 0.049640 (0.000013) (0.049640, 0.049640) 0.051031 (0.000019) (0.051031, 0.051032) Note: Ur ? Uniqueness Pair Correlation; Fr ? Latent Factor Correlation; n = 75000;