Assessing Spiritual Development: Reflections on Building a Community Measure

Measuring a complex and theologically challenging concept like spiritual formation can be daunting. This article describes a multiyear process and methodology that was used in constructing a measure that demonstrated reasonably high levels of reliability, validity, and useability. The article also describes the many challenges in developing this type of measure and strategies for overcoming those challenges. Reflections on how a measure such as this might be helpful, as well as potentially challenging, to churches and pastors are provided in the closing.

of a community of faith, we hope that we help people move forward in both their understanding of what a life aligned with God's purposes is like and encouragement to adopt in their daily lives the practices that God calls us toward in our journeys of faith. Practically, this involves helping people understand and experience transformation into a life that was modeled for us by Jesus.
But how is a pastor or ministry leader to understand where their people are in their journey of spiritual transformation? Looking out at the people attending a church service on Sunday, or at a Bible study during the week, with their smiles and looks of attention, can give us some sense of whether people are connecting with the message or teaching, but this really provides little true awareness of whether this is influencing their underlying spiritual identities or the behaviors they enact when they have left the building or in their daily lives during the week. In this same way, as a professor of child and family policy for over four decades, I could look out on a class and get some idea of whether they were understanding what I was teaching, but I really did not know what impact I was having until they took a test, wrote an essay, or talked to me in person on an oral exam. It is hard to know just where people are, except for those few who come to us and share their stories honestly and confidentially.
Likewise, it can be difficult to know how to guide a person or small group that is wanting their lives to be transformed to be more like what Jesus has taught and lived or to understand where on that journey they may be. When I was teaching and guiding graduate students, I found that many of my students struggled with their exams because they had focused on what they already knew or were good at; they paid less attention to those things they really needed to know that were outside the paradigms of their current knowledge or experience. In pre-marriage counseling, we know this to be true. Couples often avoid the topics or issues that are likely to bring tension and subsequent growth in their relationship. To respond to this need, David Olson, a colleague at the University of Minnesota, created the well-known Prepare assessment tool for these couples. 2 Pre-marriage counselors use this measure to provide insights into where a couple is currently aligned in their lives and where they need to pay attention to the changes that will help them be successful in their marriage. That assessment is widely used in pastoral counseling and has now been used by well over a million couples across the world, and it has proven to statistically predict whether marriages are likely to divorce or not. 3 My skills are in the measurement of challenging concepts and these have been applied in areas such as marital quality, 4 military family preparedness, 5 successful welfare transitions, 6 middle school engagement, 7 pastoral care competence, 8 and innovative organizational cultures, including in churches. 9 My involvement in the development of a measure of spiritual formation initially came from a request to review a spiritual maturity survey that had been used in churches in Canada. This survey was credited with helping churches better understand the spiritual needs of their people. I was asked to review the survey and I wrote detailed comments on each question. My comments were subsequently provided to the sponsor of the survey. In those comments, I noted that many of the questions on the survey had serious problems, and I doubted that the survey interpretations could be counted on as accurate. In fact, I asked for permission to use the questions in my research methods class as an example of how not to ask survey questions.
The problems I saw on that survey were common to many surveys and included: questions that were double (or triple) loaded so more than one condition was being referenced, answer choices that did not fit the lead question, answer choices that were biased or loaded in one direction, questions that were too complex for most people to answer, questions with such a high reading level that people without advanced degrees could not be expected to correctly answer them, and questions that only a theologian would be able to confidently answer. There were also problems with the response rates in many of the churches so it would be difficult to claim that the data were representative. And ultimately, this could lead to misinterpretations of the data and provide an incorrect picture of the spiritual health and maturity of these churches and their people.
This experience led me and Gary Best, 10 a Canadian pastor and former leader of the Vineyard Churches of Canada, to review measures that churches were using to assess their effectiveness in monitoring the spiritual health of the people in their communities. We found nearly all measures were focused on church organizational and cultural issues: how well the church is led and organized to carry out its work. Measures of spiritual formation, when used, were likely to be much more detailed, belief focused, or take longer to complete, so less helpful for a whole church assessment. Thus began a multiyear effort to develop, test, revise, and make available a measure of spiritual formation that became the My Faith Journey survey for churches and individuals.

Foundation building for a measure of spiritual maturity
Gary Best was well known in a variety of denominational circles and he invited a number of pastors from different denominations in the United States, as well as several theologians committed to spiritual formation, to a 3-day meeting hosted by a large Chicago-area church. The purpose was to identify the attributes that we should be able to observe in a life aligned with the practices taught by Jesus. These attributes would then be used to build a measure of spiritual maturity. I was invited as a Christian measurement specialist who had experience taking complex concepts and simplifying them into effective measures.
Underlying this meeting was the assumption that saying "yes" to the invitation of Jesus is not a simple contractual event; it is the beginning of a journey. Walter Rauschenbusch put it this way: "The Kingdom of God is not a matter of getting individuals into heaven, but of transforming our life on earth into the harmony of heaven." 11 Thus, the Christian life is a life-long realization of what our lives can be when lived fully aligned with God's purposes for us. This is a journey in which we must be fully and daily engaged. Imitating Jesus and becoming like him fully depends upon our cooperation with the grace he willingly extends toward us. Of course, Jesus made it clear that his work of transformation would progress from the inside out: a new mind and heart with new motivations would generate new ways of behaving. This assumption of spiritual formation being a journey, not an event, was the basis for defining the attributes of this life.
In this meeting, we together acknowledged that an essential aspect of knowing how to fully engage with and partner with this grace of God requires taking stock now and again of where we are in this process of formation. Examination, while sometimes seen in the world of the church as a negative or condemning activity, in scripture is highly encouraged. 12 Knowing where we are along our path means we will know how and where to call out to God for his grace and empowering so that we can break free and move ahead in our journeys of faith. Additionally, seeing what God has already done in us encourages our hearts to re-engage with him for what yet lies before us. While we never fully arrive in this life, as we trust God's process and plans for us, we become more and more fully aligned with God's purposes for our lives.
After significant time together in prayer and worship, the group decided to focus on what might be measurable in the areas of beliefs, practices, and the inner life of the Christian. Three groups were formed, and each was to develop attributes that we 11. Walter Rauschenbusch, A Theology for the Social Gospel (New York: Abington Press, 1917), 134. 12. 2 Corinthians 13:5; Psalm 139:23-24. might expect to be revealed in people's lives during the week, not just on Sunday. This resulted in seven spiritual journey dimensions, including beliefs about the character of God and the work of God's Kingdom today; the inner life of the believer in terms of closeness to and experiences with God; and actions toward others in service, in showing God's love, and inviting God's presence. Together, this produced 160 attributes that were overlapping and quite theologically complex. For example, one attribute was: "I am so convinced that God can and will guide my actions to encourage and participate with the bringing of his will and desires for this world into greater reality that I expect and look for these opportunities." There were another 159 attributes like this! Admittedly, these were pastors and theologians, not survey specialists, but clearly these were complex attributes and difficult to turn into easily understood survey questions and measures.

Creating a measure of spiritual maturity
The first task was to identify overlapping attributes and simplify the language for the average person. The reading level of the average adult in North America is about eighth grade, 13 and most survey researchers attempt to write their questions at that level of reading competence. Thus, the language of what would become questions or statements on a survey had to be substantially simplified, while retaining the original intent. Questions had to have only one focus so any questions with an "and" needed to be reduced to only one aspect of the attribute. Simplifying the language required substantial rework and time; this reduced the number of survey capable attributes to about 100.
The second task was deciding on how these would be presented on a written survey. To simplify the responses and allow people to answer questions thoughtfully but quickly, a common response pattern would be needed. We decided to turn all attributes into statements and to use frequency of time that each was believed to be true as the format for the answers. In the area of beliefs, the lead question would be "how often do you believe these to be true for you" and the answer choices would be "never true of what I believe" to "occasionally true" to "quite often true" to "most of the time true" to "always true of what I believe." For the areas of experiences and behaviors, the question was "how true is this of you now," with the same pattern of answers: "never or not yet true," "occasionally true," "quite often true," "most of the time true," and "always true." This pattern of answers has been found to be more likely to have a better distribution of responses and less likely to be biased in one direction or another. 14 13. Madeline Goodman, et al., Literacy, Numeracy, and Problem Solving in Technology-rich Environments among U.S. Adults (Washington, DC: National Center for Education Statistics, 2016). 14. Paul C. Price, Rajiv Jhangiani, and Chant A. Chiang, Research Methods in Psychology, 2nd Canadian Edition (Victoria, BC: BCcampus, 2015). https://opentextbc.ca/researchmet.
The third task was extensive pretesting of these 100 questions with focus groups from different communities. We first asked several pastors and church ministry leaders to review the questions to see if they were indeed clear in the language we had prepared. This led to some meaningful changes but even with these leaders, we found significant variations in responses, so that confirmed our answer categories. Our first focus group was at a church in the United States. This group of 25 was diverse in race, age, sex, and spiritual maturity. We had two seekers in the group as well as people at different levels of engagement in their faith. They took the survey and were asked to highlight any questions or words that gave them difficulty in any way. It took the group from 20 to 30 minutes to complete the survey. We then spent the next 2 hours debriefing each and every question. That led to a very substantial revision of the survey questions, and issues were raised that had not been raised by the pastors or our internal review.
The revised survey was reduced to 85 questions and reviewed in a focus group of 24 people from a large church in Western Canada. This too was a spiritually diverse group of men and women of different ages and backgrounds. They too took the survey and the debrief this time was for 1.5 hours. This group found additional issues with the questions and more adjustments were made to the wording and format of the draft survey. We also looked at the survey questions from the two focus groups and eliminated all questions that had no distributions on the answers; that is, all or nearly all the people answered either "never" or "always" to those questions. That left us with 70 questions for the survey, and the final focus group indicated that this would take between 12 and 15 minutes to complete; this is typically the maximum amount of time that people can be expected to devote to this task before response rates drop precipitously. 15 We also needed a general survey question of perceived level of spiritual maturity where people could self-identify where they see themselves now on their spiritual journey. This was important because some people in our focus groups, and in our churches, are not yet followers of Jesus but are spiritually curious and we wanted them to be able to participate. This would also allow us to see if the answers people give vary according to how central God is in their daily lives. So we asked them which best describes where they are in their current faith journey: (1) God is not important to them, (2) God is being explored to see if he fits in their lives, (3) a belief that God is real and they are trying to follow God in some parts of their life, (4) learning more and more to trust God in many practical aspects of life, and (5) seeing God as center of their life and committed to becoming like Jesus in all parts of life. We decided that people in the first two conditions would get a smaller and more targeted set of survey questions to answer.

Early lessons from a Church journey survey
We conducted our first survey of a church in Colorado that had about 750 people 16 and older who could take the survey. The survey was offered online through Survey Monkey, and a paper survey was offered to those for whom an online survey was not possible. After 2 weeks, 425 people had completed the survey for a very acceptable response rate of 57 percent. We analyzed the data and provided a detailed, 38-page PowerPoint-style report to church leadership that included a demographic profile of the respondents, level of participation in the church, their community spiritual maturity profile, and a detailed analysis of their survey answers according to their level of perceived spiritual maturity. 16 We debriefed the results of the survey to the pastors, ministry leaders, and elders of the church. The analysis we provided gave mean scores on each attribute question, as well as the percentage of respondents for whom this attribute was "always or mostly true" of them. The results confirmed that people who identified as having greater spiritual maturity had the highest levels of alignment on all the attributes. But the results also demonstrated that it is in the area of spiritual beliefs where all maturity groups were similar. Even the pre-Christians demonstrated high levels of agreement with the belief questions. The greatest variation was in the inner spiritual life and behavior of the those who were more mature in their journey with Jesus, something that is not as easy to discern in casual observations. There was also a significant difference in the daily practices of those who were more mature. For example, the more mature were much more likely to be generous with others, offer forgiveness and grace in practical ways, invite God's presence when praying for others, encourage others to see God's presence in their lives, and take actions against injustices when they see it.
There were practical lessons that we learned from this pilot test of the survey. First of all, the average time taking the survey was less than 15 minutes, and that met a critical objective for us. Second, the demographic results of the survey reflected well on the known characteristics of the church so that meant the response rate provided a good reflection of the community. Third, the church was happy to learn they had 4 percent who were seekers, and now had a better understanding of where these people are in their spiritual journeys, as well as the spiritual strengths and opportunities for growth in those at all levels of their people's journeys of faith. Fourth, we learned that leaders had a hard time interpreting mean scores; percentages were much more meaningful. Also, they preferred graphical interpretations to columns of numbers. Too many numbers were overwhelming, and they needed help in seeing just how to interpret the findings. Fifth, the leaders wanted an open-ended question where their people could tell them what the church could do to help them grow in their spiritual formation. We also learned that people who took the surveys enjoyed the experience but wondered if they too could find out where they are in 16. Mike Kelly of Great Loop Partners generously built the online survey tools and automated reporting programs that we used throughout this process. their own spiritual journeys. They completed the survey but had no way of getting feedback on their answers. Because the sample size was large enough, we also conducted some advanced statistical analyses of the data from this church survey. It is important to not just have a picture of the answers for leadership but to have confidence that the data and findings are statistically reliable and valid. We checked first to see if all the questions had good distributions with no unusual skew. We found that several questions, mostly in the area of beliefs, did not have good answer distributions so we decided to remove them from the survey in the future. Second, we wanted to see if the seven scales of items had good internal consistency reliability. The alpha coefficients for the scales ranged from 0.86 to 0.93, indicating good internal reliability. A factor analysis of all the items confirmed that they loaded on three factors: (1) beliefs about God and his plans, (2) identity as a person of faith in God, and (3) behaviors as an expression of faith in God. These factors confirmed the initial design of the survey attributes that came from the meeting in Chicago. This too indicated that the data were reliable in how they represented the respondent's answers.
A more advanced latent class analysis 17 was conducted to assess the predictive validity of the survey answers. This statistical procedure allows the program itself to group respondents according to their pattern of survey answers, to see if the groups are coherent in terms of their survey answers, and to reflect what would be expected from people at different levels of their spiritual maturity. This statistical analysis created four profile groupings of respondents to the survey: a spiritually immature group, a group less mature but growing in their faith, a spiritually more mature group, and a much smaller group highly centered in the spiritual lives. The analysis then looked at the ability of the questions and scales in the survey to discriminate between these groups. In every case, the scales in the survey were statistically able to discriminate between those who were operating at the different levels of spiritual maturity, thus concluding that the survey items and scales had construct and predictive validity, a critical finding for trusting the survey results. The latent class analysis also effectively discriminated the pattern of responses for males and females, with males less spiritually mature, and by age group with spiritual maturity more common among older adults. These findings would have been predicted based on previous research. 18 open-ended question at the end of the survey for people to let their church know how they could be helped to move forward in their spiritual journeys. We also asked people at the end of the survey the same five-level perceived spiritual journey status question, to see if people's perception of where they were in their journeys of faith changed after answering the now 63 journey questions on the survey. Three new churches were surveyed, two in the United States and one in Canada. All were different in their theological orientations and community context. The new report format was more graphic and that was well received. A new and separate "Comments Report" was produced that summarized the comments by gender and level of spiritual maturity. We were surprised to find that over 50 percent of the survey respondents in all the churches offered comments, and these comments and suggestions were highly regarded by the pastors and leaders of those churches.

Updating the journey survey process
We also learned that many people changed their status on spiritual maturity after answering the questions on the survey. Over 20 percent of people who reported at the beginning of the survey that they were "centered" in their life with God selected a lower maturity status after answering the questions. In fact, all the maturity levels dropped after answering the questions. We see this as a common pattern in research and in evaluations of intervention programs. People come into a survey or a program with a somewhat exaggerated sense of what they know or believe to be true of themselves. 19 In this case, the survey gave them a chance to reflect on their journey of faith and define better some of the areas of their life where God may be inviting them to grow in their walk with him.
After reviewing the experiences of the four churches, we took time to see just what was happening and to meet together to invite the Holy Spirit into our review. We had learned that pastors were pleased with what they learned but that the structure of the report around the seven spiritual journey outcomes was difficult to interpret and use in their church planning. They reported to us that this mix of spiritual beliefs, inner life processes, and behaviors could benefit from a more discrete set up spiritual journey outcomes. To do this, Gary Best and I invited a diverse group of pastors, a regional leader of InterVarsity, a Catholic scholar and researcher, a Christian psychologist and spiritual director, and several church lay leaders to a 3-day open-ended survey review process meeting. The central question before us was: how can the questions on the survey be organized to affect a better understanding of where people are in their journeys with Jesus? We had no preordained answer to that question.
The first day was spent mostly in prayer, worship, and embedding ourselves in the guiding question before us. We immersed ourselves in the Journey survey questions and we openly prayed for the Holy Spirit's guidance as to how best to understand 19. Betty Ann Carroll, Dennis K. Orthner, and Steve Day, They Don't Know What They Don't Know: Using and allow the survey questions to reflect the complexities of people's spiritual journeys. The second day was one of breakthroughs. In small groups, we realized that our journeys with Jesus often started with Trusting God's word and/or Experiencing God's presence and then flowed into Expressing God in different ways to others and Serving God as an expression of our being loved by God. This led to our uncovering those survey questions that fell into each of these faith Journey Goals.
And that led to our discovering that the questions that reflected each of these Journey Goals could be further divided into four unique Pathways for each of those Journey Goals. At the end of the third day, we had reorganized all the journey survey questions into the four Key Faith Journey Goals and their aligned questions around four Pathways to each Goal. Thus, we now had 16 practical Pathways with specific questions for each. We all felt that we had been overcome by the Holy Spirit's leading and direction. And we closed in prayer and thanks for God's presence with us. This process did not change the survey structure or questions, but it opened up the possibility for building a practical and personal feedback report for each of the people who took the survey. It also provided a structure for a much more meaningful summary church report with more easily interpreted findings and guidance for how a church might use the data.
The structure of the new individual and church feedback reports now followed this new model of the My Faith Journey Goals and Pathways: Trusting God: four pathways to that Goal, including questions on believing God's story, receiving God's love, having a personal identity in God, and trusting God when challenges come. Experiencing God: four pathways to that Goal, including questions on responding to God's leading, regularly communicating with God, growing in personal faith, and engaging in worship. Expressing God: four pathways to that Goal, including questions on expressing God's love to others, practices in faith toward others, journeying with others, and reconciling and forgiving others. Serving God: four pathways to that Goal, including questions on actively partnering with God, stewarding resources, showing compassion, and acting justly with the weak and powerless.

Implementing the new journey survey and reports
We created a new personal feedback report for everyone who took the My Faith Journey survey. The data from their online survey for each person was kept completely private but when they finished the survey, respondents were given the opportunity to print out a personal report. This gave them some guidance as to how to interpret the findings, and a graphical picture of their progress on the 16 Pathways, including areas where they were already strong in their faith journeys, and those areas where God is inviting them to grow. The focus was on positive, invitational language rather than confrontational or negative interpretations. In addition to the graph, the report included personal and customized feedback on each of the 16 pathways, including guidance as to how they might grow their faith in those areas where God might be calling them to new directions in the spiritual life. 20 This same approach was taken with the updated church summary report. It too provided more graphical and personalized information on the spiritual journeys of their people. In this report, we now categorized the levels of spiritual maturity by the survey answers to the end-of-the-survey spiritual maturity question, instead of the beginning one since we had learned that this provided a better reflection of where people currently are in their journeys with God.
We now invited additional churches in the United States, Canada, and England to try out this new survey and reporting process. We also created a new online survey process that allowed us to customize our surveys and reports in ways that Survey Monkey and similar online systems did not permit. We surveyed small churches as well as large churches, from different denominations to see if the process was working and helpful. Two churches had used the older format and were very pleased to see the meaningful data from the new report process. The personal feedback reports were very well received and led to a number of churches using those reports in small group discussions. One church even created a 5-week small group guide for their people to use for discussing their individual findings together. 21 Here are a couple of encouraging comments from pastors who used the new version of the survey and reports: Not only has the process launched a church-wide conversation about personal growth, but our leaders are also more energized than ever about pouring their time into pastoring and mentoring. What once looked like a sociological growth problem to be solved has now come into focus as an exciting opportunity to welcome people fully into life Jesus offers. (Michigan, U.S. pastor) The spiritual journey survey was a breath of fresh insight for our church. It confirmed unquantified trends, revealed valuable insight into areas of necessary teaching, and provided accurate feedback that would have been otherwise very difficult to determine. The results of our survey were both very encouraging and refreshingly sober. Thanks to this, a relevant way forward is emerging that will inform everything from preaching, small groups, outreach, Sunday worship and other areas. (New Brunswick, Canada pastor) 20. The online My Faith Journey survey can now be taken, and a report provided, to people who are not part of a faith community effort. A link to this survey can be provided by the author. 21. The My Faith Journey small group guide is available from the author.

Reflections on measuring our spiritual journeys
First of all, creating a measure of spiritual development is a process of humility and compromise. No set of brief questions can capture the fullness of what God is calling us to be and to express in our lives. Any set of metrics that tries to do this must "keep it simple" and focus on those attributes that are given the greatest attention in the teachings of Jesus. And I have learned over time that surveys that take more than 10-15 minutes are less likely to be answered, and those people who do answer longer surveys will be different from those who do not. So a brief survey that gives a partial but still representative picture of our journeys with Jesus can be helpful, both to us an individuals, and collectively to us in communities of faith. Our experience is that churches value having a way to gauge the diversity of ways that people are growing in their faith and seeing those areas of spiritual formation that may need greater attention in teaching, counseling, ministries, and preaching.
Second, we have learned that measuring spiritual formation can bring both hope and tension into the leadership of a church. Hope comes from creating a church-wide collective awareness of areas of formation that deserve more attention and growth. Most churches who conducted the journey survey picked two or three key findings around which they could plan and devote attention over the next year or two. There was excitement around doing this and working together toward some measurable goals. But change, even the anticipation of change, can also bring tension. Sometimes being naive about the needs of the congregation for spiritual growth feels better than being more fully aware of the spiritual practices that are being neglected. In fact, we talked with church leaders who sincerely worried about being held accountable for "poor scores" on the journey survey among their people. Rather than seeing this as an invitation by God to grow those areas of the church that deserve more attention, they saw this as a test of their competence and chose not to participate.
Third, there has to be broad leadership commitment to the process of church-wide spiritual formation for a survey like this to succeed. The senior pastor has to fully engage in the process from the very beginning and lead or co-lead along the way. We found that a church's survey response rate was influenced substantially by the amount of attention given to it by the pastor from the pulpit and in communication with their people. A spiritual maturity survey of a church typically takes about 8-10 weeks of leadership engagement. This includes 4 weeks of dedicated planning for conducting the survey, 2 weeks of promoting and collecting the survey data, and then 2-4 weeks of careful leadership reviews of the findings and making plans for going forward. This can feel daunting in a busy church calendar so the resolve to learn the unknown and grow a people who represent Jesus more fully in their lives has to be a high priority for leadership.
Fourth, employing a set of questions and measures to gauge our levels of spiritual maturity can upset our personal or collective sense of spiritual well-being. We saw this on our survey when we asked people at the beginning, before they answered any specific questions, where they saw themselves on their journeys of faith in God. In most of our churches, nearly half of the people reported at the beginning that their lives were "centered around God." Our analysis found that this was primarily based on their agreement with the core beliefs of Christianity, not on the spiritual practices that should flow from those beliefs. After answering the survey questions, that percentage of people seeing themselves "centered" dropped significantly. Just as using a good set of metrics in pre-marriage counseling can reveal that not all is well in a relationship, a set of spiritual maturity metrics can reveal gaps in our journeys of faith that need to be carefully turned into a positive and welcoming invitation to foster new practices and lifestyles. Committing to this invitation to grow our lives more fully aligned with God's purposes takes time and Holy Spirit-led guidance along the way.
Fifth and finally, the findings of any spiritual growth survey must be complemented with the voices of the people. That is why the Journey Survey Comments Report was such a valuable asset to church leadership; they could read what their people wrote in answer to the question: "How would you like your local church to help you walk closer to Jesus?" People took this opportunity seriously and over half of the survey respondents wrote helpful comments and recommendations that they hoped their pastors and leaders would listen to as they reviewed the more quantitative survey report. Many pastors held church-wide reviews of the survey results and invited people to comment on what they saw and how they felt about the results. This led to broader conversations and even agreement in the church about spiritual formation and gave pastors insights into the practical ways that this can be nurtured in their community. Listening to both the survey data and the voices of the people can together provide a powerful set of tools and motivations upon which to build a spiritually healthy community.