3 People, capability & collaboration
3.1 Team capability
Typical health data science projects require a multidisciplinary team. The involvement of different roles will likely fluctuate through the different stages of a project lifecycle.
Some individuals will have multiple skills, but few will have everything they need. Commonly needed skills in the team, or accessible through other organisations, include:
- subject matter expertise in data
- machine learning
- system design
- user interface design
- clinical
- governance
- consumer/patient and specifically impacted communities
- legal and privacy
- ethics
- implementation and change management.
You’ll often be involved in multiple projects, so it’s useful to share expertise across projects.
It’s particularly important to allow plenty of time for clinical input to data understanding and preparation for data science.
Defining input features in clinically relevant ways is particularly important for health data science. For example, ‘cancer’ might be an important input at an individual level; does this mean currently active cancer? Within the last five years? Any exclusions? Time needs to be allocated to work through and validate these definitions with clinicians.
Advisory groups are a good way to elicit feedback. Early in any project try to meet with a group of experts and discuss your research plan. Experts will always have different perspectives on the research and the context in which it will be used (see End-user engagement below, which also covers Māori engagement and co-design).
3.2 End-user engagement
You should engage end users (usually consumers and clinicians) early in co-design, to understand how outputs can be tangible for those who will use them or be impacted by them (see Transparency, interpretability, and explanation). It’s important to understand workflows and where tools and/or models may be used. End-user engagement will also help to drive IT implementation if the benefits are clearly articulated to the appropriate stakeholders so work can be prioritised.
3.2.1 Impacted customers/patients/groups
Data scientists are unlikely to have the correct context or cultural awareness to fully grasp what the data is telling them. This includes Māori and other ethnicity groups, consumers, and perspectives that cover age and ability ranges.
Any recommendations for changes or improvements should be interpreted through the lens of the groups they are likely to affect, acknowledging the principle of “nothing about us without us”.
HQSC’s Code of expectations for health entities’ engagement with consumers and whānau is a useful framework to bear in mind.
3.2.2 Clinicians
Co-designing from the start and having a clinical champion/sponsor to ensure that developments can be incorporated in existing workflows so they can be used will set you up for success (see Operational deployment).
Clinical engagement often makes or breaks a project - both to ensure efficacy and to advocate and promote use of the model in practice. In general, clinicians will adopt tools and models that solve a genuine problem, fit into their workflow patterns, and save them time or enhance safety. It is worth thinking about the different professions that might be involved (medical, nursing and allied professions) as well as specific skills that some professions bring such as in epidemiology, public health and biostatistics. There are also other groups involved in delivery of care that are not professionals but may have useful insights (eg. data analysts, administrators, commissioners).
Interpretability (which is defined as understanding the reasoning behind predictions and decisions made by the model) is also key. Important stakeholders are unlikely to have lots of time to learn about your work, so displaying it in an easy-to-understand way is essential. You shouldn’t assume that everyone will interpret outputs in the same way, and understand what action is then required. Keep in mind that prospective customers may not be technologically or mathematically savvy, too.
3.3 Collaboration
Agree collaboration will be be supported across different organisations (who may all be using different software and systems and have limitations around what can be used).
Role definition, responsibilities
Regular check-ins/stand-ups are helpful
Be mindful of individuals’ schedules and availability, particularly clinicians.
3.4 Measuring success
Be clear what success looks like. This will be different for research vs. implementation.
- Are you looking to learn/measure something specific?
- Are you expecting secondary or downstream benefits
Making sure the bigger picture is kept in mind - not getting lost in the detail in a way that doesn’t add value
Have you reached the point of diminishing returns for investing further in model development?
If you have measurable benefits as an objective, you should develop a benefits realisation plan.
3.5 Project management
Data science is an iterative process of development. You’ll progressively learn more about the data, relationships in the data and how effective modelling is.
Good project management or product development practices should be applied to data science projects, while accepting that these projects are often experimental or exploratory in nature. Some important project management considerations include:
Data issues flow ‘downstream’ and can impact every other part of a project. Ensure that sufficient time is allocated upfront to review and correct data quality (this also often happens in cycles - the analysis reveals quirks in the data that can be explored further and corrected)
Project management should strive for continuous visibility, demonstrable progress. How are we tracking against timelines, budget, and the desired outcome?
Create open feedback channels where possible. Think about how to develop mockups or prototypes as early as possible for feedback - can you start with a very simple model, Excel dashboard or static design to help ensure that what you are developing will deliver value?
Healthcare data science projects will often span multiple organisations
Documenting failures and lessons learnt helps prevent repeating mistakes.
Use technology for progress tracking. Some options include:
- Jira,
- a spreadsheet-based activity tracker,
- Trello board,
- emailed summaries of actions and next steps
3.6 Governance
Good governance is fundamentally concerned with the value and risk of a project, and must be established to ensure accountability and oversight. Developing policies and procedures can help you manage risk and clearly articulate principles of accountability, transparency, ethical use, privacy, and consent.
You should define and tailor a governance approach based on the needs of your own organisation.
Usually a data science project is undertaken within a wider programme of work, which has its own governance structure and processes. Occasionally, more substantial initiatives will require their own standalone governance.
Governance may be applied at different levels, such as governance of data (inputs) or governance of models (product/output). People involved in model governance should come from diverse demographic and technical backgrounds, including perspectives of consumers and/or those who are impacted by the outputs of your work. People involved in data governance need to have an understanding of data flows - how data is captured, stored and used.
Questions to ask:
- Are ethics applications required?
- What existing governance groups and/or processes would need to be involved?
- Which policies need to be followed?
- What level of documentation is required throughout?
- Will this be shared publicly? If so, how?
- Who is responsible for signing off?
- Is a new governance structure or process required here?
- What maintenance and/or ongoing review may be required? The NICE Evidence Standards Framework for Digital Health Technologies has a helpful list of points to consider.
There are many resources for governance groups and executive teams who are looking to write data policies and procedures. These include:
New Zealand Data and Information Management Principles (Digital.govt.nz): The government’s open data policies and best-practice guidance for agencies managing how data is stored, published and used
National Ethical Standards for Health and Disability Research (NEAC): These standards set out the ethical requirements for researchers, health service providers and disability service providers and apply whether or not research or quality improvement activities require review by an ethics committee
Privacy Act 2020: the legal basis for NZ organisations to work with data safely - also see Ethics & privacy
Health Information Privacy Code 2020 - also see Ethics & privacy
HISO 10064:2017 Health Information Governance Guidelines (Manatū Hauora Ministry of Health): Guidance to the health and disability sector on the safe sharing of health information
New Zealand Government Open Access and Licensing framework (NZGOAL) (Data.govt.nz): For those who work for a government agency and want to enable appropriate re-use of your agency’s material by licensing its copyright works or releasing non-copyright material (such as open data) for re-use
A Path to Social License: Guidelines for Trusted Data Use (Data Futures Partnership): August 2017 summary with continuing relevance around what New Zealand people expect from guidelines for data use and sharing
Layered model for AI governance(Harvard University): A conceptual framework for thinking about governance for AI.
Lessons learned from developing a COVID-19 algorithm governance framework in Aotearoa New Zealand: Practical considerations from a governance group
Algorithm Charter (Data.govt.nz): A set of principles which have been signed off by most NZ government organisations (Manatū Hauora Ministry of Health)
Health and Disability Ethics Committees: the health organisation responsible for reviewing and approving research using health data and/or health system users - also see Ethics & privacy
Te Mana Rauranga: a network of experts who have collected principles for Māori data sovereignty - also see Data sovereignty
Manatū Hauora Ministry of Health - Emerging Health Technology Advice & Guidance
The Law Foundation report on government use of AI in NZ (Colin Gavaghan et al. 2019)
Trustworthy AI in Aotearoa: AI Principles report from AI Forum (AI Forum 2020)