Weljo Collaboration
Role:
R Quantitative Analyses, Experiment Facilitation & Stakeholder Delivery
Overview:
During a 5-month collaboration, my GIT Lab team partnered with Weljo to provide complimentary insight into their onboarding process and overall user experience.
Weljo is a tool designed to foster camaraderie among remote employees. We shared valuable insights from the collected data with stakeholders over multiple meetings.
Background
Beginning in the summer of 2023, Claremont Graduate University's UX lab team conducted a study looking at the impact of gameplay on hybrid and remote teams. They explored common team challenges that arise from remote work, and the role of online team-building games.
I joined this Team-Building research team in October 2023 when a collaborative effort was established with Weljo to expand on remote team-building exercises through exploratory research—the partnership utilized mixed-method survey data to deliver insights to stakeholders.
Who is Weljo:
Weljo provides software that integrates into digital workspaces (such as Slack) and provides unique activities to do with team members, from ice-breaking games to meditation.
They aim to promote healthy remote work cultures by encouraging group activities while building connections and collaboration. Since our lab operates remotely and is broken up into teams, we utilized our lab members to gather data on the impact of Weljo's activities.
Methodology
The Weljo collaborative team conducted both quantitative and qualitative analyses, with myself being part of the quantitative side for both experiments.
Quantitative analyses were conducted in R through t-tests to determine differences in participant sentiment pre and post-treatment.
Survey:
The development of survey items was a collaborative effort of all teammates past and present.
Designed to gather participant sentiment on how their team was doing across 4 dimensions:
(1) Trust and Collaboration, (2) Inclusion
(3) Psychological Safety, (4) Effective Communication.
9 Likert-scale questions were used with a 7-point scale and were sourced from existing peer-reviewed and published scales. For responses, 1 was "Never" while 7 was "Always"
Experiment:
Experiment 1:
Individual team participation in an unmoderated game via integration through Slack.
5 Teams consisting of 2-5 people each.
Sample size: n = 15
Experiment 2:
Weljo provided a moderator for the entire lab participation.
1 entire group
Sample size: n = 15
Both Experiments:
Participants first took a 3 to 4-minute pretest survey about team cohesion before conducting a 15-minute Weljo break followed by a posttest identical to the survey they took beforehand.
During this posttest, 6 additional qualitative questions were included to allow for more open-ended feedback to be given.
Participant groups were free to choose which Weljo activity they participated in.
Data Analysis
T-tests were conducted in R utilizing practiced scripts.
Results:
Two statistically significant differences for experiment 1
One statistically significant difference for experiment 2.
Question 5 Descriptives in R
Question 9 Descriptives in R
Insights
Experiment 1
For experiment 1, our team generated key insights from an unmoderated individual team session in Weljo.
Quantitative:
Insight 1:
Participants scored higher in affective trust after the activity, measured by the statement in Question 5: “I feel supported by my team when facing challenges in my work.”.
Mean increased from 5.6 to 6.2
Insight 2:
An increase in team cohesion was supported by the increase in participant satisfaction with their team’s commitment, measured by the statement in Question 9: “I am happy with my team’s level of commitment to tasks.”
Mean increased from 5.07 to 6.13
Other:
There was no detectable change in any of the other seven survey items. You can view the increase in means from pretest to post-test responses to Question 5 and Question 9 below:
Experiment 2
For experiment 2, our team generated four key insights from a moderated complete lab session in Weljo.
Quantitative:
Results for experiment 2 were similar to experiment 1, which is important to note since the main difference between the experiments is the external moderator.
Insight 1:
Our first insight was that participants reported a greater sense of alignment with their team’s goals. This was measured by the statement in Question 7: “My team and I are working toward a common goal."
Mean increased from 5.67 to 6.00
Qualitative:
Insight 3:
Repeated themes across participants were found. The positive impact of Weljo was felt the strongest on newly formed teams and also on those who did not have a connection already.
Furthermore, team leaders specified that the Weljo activity sparked meaningful conversation. A majority of participants reported interest in continued Weljo use.
Insight 4:
Despite the improvements to team cohesion, affective trust, and overall enjoyment of the team bonding, the Weljo activities were not without their pain points.
Onboarding was not clear for all users, as some reported missing the join button on the invites that were sent out. As a result, this required team leads to resend invites to get everyone together again.
Because this was outside of the scope of our research, we do not know the exact cause. Further research could help us understand this issue better.
Qualitative:
Insight 2:
Participants saw value in using Weljo for team-building activities. About 83% of the participants found the moderated session fun, interesting, and unique, with 44% wanting to see changes.
Insight 3:
There were some concerns regarding the moderator and other clarity issues with instructions. 25% of participants had negative feelings towards the moderator and another 25% also reported feeling lost or confused.
Insight 4:
Despite concerns, 61% of participants still said they would use Weljo again in the future!
Even more so, 80% of the participants felt it would be a great way to bond with their team in the future with responses indicating positive effects such as relaxing or cutting loose with teammates.
Conclusion
Stakeholder Deliverables:
2 stakeholder meetings were conducted, one for each experiment. After the conclusion of the second meeting, we received positive feedback and were invited to further work with Weljo's real-world clientele:
Notable Quote:
"I see value in your research. For the next steps, we have new clients and maybe if you would directly participate with our clients you can see more real data of larger teams who are using the product."
Impact:
When we took a step back and looked at the big picture of these experiments, we found significant improvement in aspects of team cohesion within our lab.
Both quantitatively and qualitatively, we found that using Weljo made an overall positive impact on our team members. While there may be some room for improvement, we feel Weljo is a useful tool to use if your organization utilizes a unified platform like Slack, Teams, etc.
Although I was not part of the team during this time, this research builds upon our prior blog post that team-building games and activities can be a great way to mitigate common challenges faced at work, and Weljo is certainly on the list of effective activities to consider implementing.
-------------------------------------------------------------------------------------------------------------------
Q & D Summary:
Our Weljo collaboration lasted over two studies, with stakeholder presentations for findings delivered upon analyses. Both studies were run among the GIT Lab teams. The 1st experiment was an unmoderated activity between each lab group, while the 2nd experiment was conducted as a single lab group while being run by a Weljo-provided moderator.
A pretest-posttest survey design was given with additional open-ended questions delivered after experimentation.
My role in the analysis was on the quantitative end which derived a few major insights: (1) A higher score in affective trust and (2) an increase in team cohesion during the first experiment. For experiment 2, (3) participants saw a greater sense of alignment with team goals.
The qualitative analyses were true to capture meaningful data as well. For experiment 1, the onboarding process was not clear for all users. Since this required further investigation beyond the scope of our research, the causes for these issues are not exactly known. For experiment 2, a quarter of participants had negative feelings towards the moderated activity. This was coupled with a quarter of participants feeling confused during this session.
Despite analyses highlighting some areas of improvement, our research team still received positive feedback and an invitation to further work with Weljo's real-world clientele.
A summarized post of our collaboration and results was posted to the Claremont UX Research Laboratory blog which can be found here.
Lessons Learned
R Script: At the time (Fall 2023), I was still relatively new to utilizing R for analysis. This practical example outside of the classroom helped highlight areas of improvement necessary to become more comfortable with the software.
Flexibility: This was my first real-world experience with UX. It allowed me to practice flexibility regarding new tasks and deadline changes.
Personal: A personal area of improvement of mine is the unfounded obligation to do more than required. Having a reliable team allowed me to practice role discipline but also focus on refining my area of collaborative effort.
Above is my excerpt from the first stakeholder meeting.
