Improving the player experience on multiple projects across time zones.
This is an ongoing project that is my main priority as an Intermediate User Researcher at NetEase Games Montreal. NetEase Games Montreal has recently rebranded to Highdive, and while there are new IPs in development, my work has mostly focused on assisting co-development games that are being developed by other studios under the NetEase umbrella. These co-development games include the recently announced Ananta, as well as FragPunk.
In my role, I was tasked with moderation, analysis, and reporting for in-person playtests for these previously unannounced projects, as well as leading specific research projects. This entails being the point of contact between the Insights team that I am part of and the development team, as well as developing the broad research questions for the test, drafting survey questions, leading recruitment, creating the timeline for the day-of test, outlining analysis goals, and writing and presenting the report. As such, I have had the chance to do the work of a Senior User Researcher: skilling up and taking on more responsibility on these projects.
Objectives:
Unfortunately, due to the sensitive nature of the information and because these projects are still in development, I cannot go into detail on the objectives of the research specifically.
Process:
The process of conducting user research on a co-development project is relatively similar to in-house, with a few exceptions. Initial requests for a research study are usually based on key milestones, such as FPP (first playable prototypes) or moving from pre-production into production phases of development. Alternatively, requests could be made for testing specific mechanics or portions of the game, such as button layout, narrative and story, or newly implemented combat mechanics.
These requests are given to the director or manager, and they create the project team, usually consisting of 2-3 user researchers, and sometimes 1-2 data analysts if there is a need. Throughout my time at NetEase, I have gradually accumulated enough knowledge and expertise to lead some of these projects, which means that I engage in more planning and delegation, as opposed to strictly recruitment, moderation, and analysis.
As an example, when leading a research project on one of our co-development projects, I was the point of contact for the development team and producer, where I would answer any questions from them, as well as verify test requirements and goals, and provide consistent progress reports on tasks. This allowed us to work efficiently even across 12 time zones, as we had one asynchronous point of contact responsible for stakeholder management. While I cannot go into specifics, while leading a research project on one of our co-development projects, I was able to:
- Develop the test goals and research questions the team wanted to answer
- Set recruitment criteria and write a recruitment screener to be sent to potential participants
- Create a list of tasks that both myself and the other researchers would need to accomplish
- Create a draft timeline of the test, including sections of the game to play, and allotting time for each section which included surveys and breaks for players
- Communicate with the development team about any notable changes in the build since our last test, and to validate our research goals, research questions, timeline, and the development team’s highest priority issues, while also keeping the development team up-to-date on the progress of the research
- Delegate tasks to the other user researchers, to efficiently complete tasks based on the timeline and priority of each task
- Create a moderation guide for researchers during the in-person test of the game, highlighting anything to look for in gameplay and to outline each section of the test
- Build an analysis template/guideline to discover high priority issues quickly, as well as create topline answers to the most important research questions as quickly as possible
- Iterate on a report template and adjusting the template to meet the needs of the development team while also altering the template to best illustrate the data and analysis we have gathered
- Lead the presentation, focusing on legibility and understanding as the co-development team’s first language is not English (this means focusing on videos to highlight any issues, reducing language complexity in the presentation and report, as well as offering the report in advance of the presentation to give the co-development team time to review and prepare any questions they may have)
Key Learnings:
- Asynchronous communication requires a lot of consideration, as typical turnarounds for messages are 1 business day at least. This means that clear, direct questions are a necessity, and making sure every question gets answered requires employing tactics such as enumerating each question or consolidating multiple asks into one message.
- Building relationships with overseas partners has its own set of challenges. Working with co-development partners, we want to make sure we are useful and adding value to the games we do research on. This means being understanding when communication takes longer than expected, or when bugs have not been ironed out and require a new build to test on.
- Doing great work means relying on others. During many points in leading these research projects, I was unsure about how best to move forward – do I follow my own best practices? Do I go with standards that have been set before within the team? Do I go with what is the most time efficient? I am extremely lucky that I had support from an amazing research team that helped me develop my skills and grow confidently.
Reflections:
What went well – I had an amazingly supportive team that allowed me to get outside of my comfort zone and try new things within my research. It also allowed me to flex my abilities, especially in terms of project and stakeholder management, as well as translating my doctorate research skills into what is needed within games user research.
What could be improved – Keeping everyone in the loop, including the research project team, the research team at large, managers and directors, as well as the co-development partners turned out to be quite difficult. Setting clear timeline expectations and making sure adequate time is set aside for internal research review is absolutely vital to avoid any unexpected last-minute edits.
Overall, I believe that my work continues to have a hugely positive effect within the ongoing work at Highdive. I am incredibly proud of the work that I have been able to accomplish so far, and I am excited to continue on my journey to improve the player’s experience within these amazing game worlds.
