User login
Methodological Progress Note: Group Level Assessment
Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.1 It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.7 GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).8
Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.1 Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.1
APPLICATION OF GLA
Stakeholder Recruitment
GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.8
Session Logistics
Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.5 Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.
Prompt Development
Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.1 This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. 8 Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.
Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.
GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.
Description of the Seven Steps in GLA
In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.
In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.
Data Analysis
Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.
Complementary Data and Synthesis
Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.8
STRENGTHS OF GLA
Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).1 When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.
LIMITATIONS OF GLA
Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”1 Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.
WHY DID THE AUTHORS USE GLA?
Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.
Disclosures
The authors have nothing to disclose.
1. Vaughn LM, Lohmueller M. Calling all stakeholders: group-level assessment (GLA)—a qualitative and participatory method for large groups. Eval Rev. 2014;38(4):336-355. https:// doi.org/10.1177/0193841X14544903.
2. Gosdin CH, Vaughn L. Perceptions of physician bedside handoff with nurse and family involvement. Hosp Pediatr. 2012;2(1):34-38. https://doi.org/10.1542/hpeds.2011-0008-2
3. Graham KE, Schellinger AR, Vaughn LM. Developing strategies for positive change: transitioning foster youth to adulthood. Child Youth Serv Rev. 2015;54:71-79. https://doi.org/10.1016/j.childyouth.2015.04.014
4. Schondelmeyer AC, Jenkins AM, Allison B, et al. Factors influencing use of continuous physiologic monitors for hospitalized pediatric patients. Hosp Pediatr. 2019;9(6):423-428. https://doi.org/10.1542/hpeds.2019-0007
5. Vaughn LM, Jacquez F, Zhao J, Lang M. Partnering with students to explore the health needs of an ethnically diverse, low-resource school: an innovative large group assessment approach. Fam Community Health. 2011;34(1):72-84. https://doi.org/10.1097/FCH.0b013e3181fded12
6. Vaughn LM. Group level assessment: a large group method for identifying primary issues and needs within a community. Sage Journals. 2014;38:336-355. https://doi.org/10.4135/978144627305014541626
7. Vaughn LM. Psychology and culture: thinking, feeling and behaving in a global context. 2nd ed. New York, NY: Taylor & Francis; 2019.
8. Choe A, Unaka N, Schondelmeyer AC, Bignall, RW, Vilvens H, Thomson J. Inpatient communication barriers and drivers when caring for children with limited English proficiency [published online ahead of print July 24, 2019]. J Hosp Med. https://doi.org/10.12788/jhm.3240.
Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.1 It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.7 GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).8
Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.1 Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.1
APPLICATION OF GLA
Stakeholder Recruitment
GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.8
Session Logistics
Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.5 Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.
Prompt Development
Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.1 This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. 8 Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.
Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.
GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.
Description of the Seven Steps in GLA
In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.
In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.
Data Analysis
Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.
Complementary Data and Synthesis
Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.8
STRENGTHS OF GLA
Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).1 When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.
LIMITATIONS OF GLA
Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”1 Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.
WHY DID THE AUTHORS USE GLA?
Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.
Disclosures
The authors have nothing to disclose.
Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.1 It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.7 GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).8
Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.1 Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.1
APPLICATION OF GLA
Stakeholder Recruitment
GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.8
Session Logistics
Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.5 Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.
Prompt Development
Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.1 This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. 8 Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.
Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.
GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.
Description of the Seven Steps in GLA
In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.
In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.
Data Analysis
Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.
Complementary Data and Synthesis
Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.8
STRENGTHS OF GLA
Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).1 When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.
LIMITATIONS OF GLA
Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”1 Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.
WHY DID THE AUTHORS USE GLA?
Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.
Disclosures
The authors have nothing to disclose.
1. Vaughn LM, Lohmueller M. Calling all stakeholders: group-level assessment (GLA)—a qualitative and participatory method for large groups. Eval Rev. 2014;38(4):336-355. https:// doi.org/10.1177/0193841X14544903.
2. Gosdin CH, Vaughn L. Perceptions of physician bedside handoff with nurse and family involvement. Hosp Pediatr. 2012;2(1):34-38. https://doi.org/10.1542/hpeds.2011-0008-2
3. Graham KE, Schellinger AR, Vaughn LM. Developing strategies for positive change: transitioning foster youth to adulthood. Child Youth Serv Rev. 2015;54:71-79. https://doi.org/10.1016/j.childyouth.2015.04.014
4. Schondelmeyer AC, Jenkins AM, Allison B, et al. Factors influencing use of continuous physiologic monitors for hospitalized pediatric patients. Hosp Pediatr. 2019;9(6):423-428. https://doi.org/10.1542/hpeds.2019-0007
5. Vaughn LM, Jacquez F, Zhao J, Lang M. Partnering with students to explore the health needs of an ethnically diverse, low-resource school: an innovative large group assessment approach. Fam Community Health. 2011;34(1):72-84. https://doi.org/10.1097/FCH.0b013e3181fded12
6. Vaughn LM. Group level assessment: a large group method for identifying primary issues and needs within a community. Sage Journals. 2014;38:336-355. https://doi.org/10.4135/978144627305014541626
7. Vaughn LM. Psychology and culture: thinking, feeling and behaving in a global context. 2nd ed. New York, NY: Taylor & Francis; 2019.
8. Choe A, Unaka N, Schondelmeyer AC, Bignall, RW, Vilvens H, Thomson J. Inpatient communication barriers and drivers when caring for children with limited English proficiency [published online ahead of print July 24, 2019]. J Hosp Med. https://doi.org/10.12788/jhm.3240.
1. Vaughn LM, Lohmueller M. Calling all stakeholders: group-level assessment (GLA)—a qualitative and participatory method for large groups. Eval Rev. 2014;38(4):336-355. https:// doi.org/10.1177/0193841X14544903.
2. Gosdin CH, Vaughn L. Perceptions of physician bedside handoff with nurse and family involvement. Hosp Pediatr. 2012;2(1):34-38. https://doi.org/10.1542/hpeds.2011-0008-2
3. Graham KE, Schellinger AR, Vaughn LM. Developing strategies for positive change: transitioning foster youth to adulthood. Child Youth Serv Rev. 2015;54:71-79. https://doi.org/10.1016/j.childyouth.2015.04.014
4. Schondelmeyer AC, Jenkins AM, Allison B, et al. Factors influencing use of continuous physiologic monitors for hospitalized pediatric patients. Hosp Pediatr. 2019;9(6):423-428. https://doi.org/10.1542/hpeds.2019-0007
5. Vaughn LM, Jacquez F, Zhao J, Lang M. Partnering with students to explore the health needs of an ethnically diverse, low-resource school: an innovative large group assessment approach. Fam Community Health. 2011;34(1):72-84. https://doi.org/10.1097/FCH.0b013e3181fded12
6. Vaughn LM. Group level assessment: a large group method for identifying primary issues and needs within a community. Sage Journals. 2014;38:336-355. https://doi.org/10.4135/978144627305014541626
7. Vaughn LM. Psychology and culture: thinking, feeling and behaving in a global context. 2nd ed. New York, NY: Taylor & Francis; 2019.
8. Choe A, Unaka N, Schondelmeyer AC, Bignall, RW, Vilvens H, Thomson J. Inpatient communication barriers and drivers when caring for children with limited English proficiency [published online ahead of print July 24, 2019]. J Hosp Med. https://doi.org/10.12788/jhm.3240.
© 2019 Society of Hospital Medicine