Image by Olga Kononenko

HRSA Performance Measures

Connecting the Dots: Entwining the Organization’s Mission with Its IT Capabilities at HAB
Overview

Innovate! (Innovate), in partnership with Customer Value Partners (CVP), led a multi-year engagement that focused on conducting Return on Investment (ROI) analyses and standing up a performance measurement hierarchy to evaluate IT investments for the Health Resources and Services Administration (HRSA) HIV/AIDS Bureau (HAB).  Ultimately, the objective of this initiative was to stand up a repeatable mechanism to guide HAB toward the IT investment decisions that maximize value for the Bureau and serve its mission of supporting the best patient care possible. 

The success of this work led to the 2021 Fed Health IT Innovation Award -- and HAB leadership has since welcomed back the analysis and portfolio management teams to continue their contributions to the HAB IT landscape in an expanded, Innovate-led effort.  To date, HAB leadership has leveraged the recommendations coming out of these efforts to stand up governance around its IT portfolio -- and to issue funding decisions based on the business needs that have emerged throughout this work.

About the Organization

The Health Resources and Services Administration (HRSA), an agency of the U.S. Department of Health and Human Services (HHS), is the primary federal agency for improving healthcare to people who are geographically isolated or those who are economically or medically vulnerable.

 

HRSA programs help those in need of high-quality primary healthcare, people with HIV/AIDS, pregnant women, and mothers.  HRSA also supports the training of health professionals, the distribution of providers to areas where they are needed most, and improvements in healthcare delivery.

The HIV/AIDS Bureau (HAB) administers the Ryan White HIV/AIDS Program (RWHAP) and the Global HIV/AIDS Program by administering grants and managing grant performance in support of nationwide and worldwide efforts to provide healthcare that furthers improved patient outcomes.  Together, HAB program and operational divisions play a critical role in helping recipient clinics to diagnose, treat, prevent, and respond to patient needs.  Essentially, HAB is committed to supporting grant recipients in their efforts to improve the quality of life for all patients.

Effort Approach

Key Objectives

  • What

    • Develop performance measurement framework to evaluate how well IT investments align to HAB organizational priorities

    • ROI analysis to extract insights from chosen projects (6 analyses completed during FY2019 – 2021)

  • How

    • Info Gathering: stakeholder interviews, focus groups, surveys, existing data from relevant efforts

    • Collaboration with project leads, impacted users, HAB leaders, and external partners

    • Quantitative and qualitative analysis and alignment to HAB organizational performance hierarchy

  • Why

    • Foundation for tying project performance to HAB priorities and mission

    • Understanding of key successes and lessons learned from examined projects

    • Distillation of recommendations for future action

Customized Methodologies
  • ROI Methodology

  • Explain steps in performing ROI

  • Speak to strategic alignment of ROI to HAB and regulatory priorities

ROI Methodology Steps...
roi-methodologiey steps_edited.jpg

1. Define Project Scope: SharePoint 2.0 and Extensions
The scope of the SharePoint 2.0 enterprise social network is comprised of 18 separate applications. The individual scopes of these 18 applications were shaped by the expressed needs of their respective staff user communities and were likely limited by project budget constraints.
 
An enterprise social network, such as HAB’s SharePoint 2.0, will grow and mature naturally to address new opportunities identified by its user community.  This ROI methodology, thus, is designed to provide both a rigorous “snapshot” assessment of SharePoint 2.0 performance, as well as the basis for making approximations of performance that may result from other SharePoint investments. 
 
2. Define Project Costs and Benefits
Project costs are defined as costs incurred by HAB and HRSA to complete SharePoint 2.0 application development, deployment, and maintenance.  Project cost savings are any subsequent dollar reductions in system maintenance costs accruing to HAB or HRSA from the elimination of service redundancies. [1]
 
Defining Project Benefits

defining-project-benefits.jpg

Studies of enterprise social networks[2] have found a set of recurring organizational benefits.
 

3. Estimate Project Costs and Benefits
This ROI methodology combines three major approaches that are commonly used for estimating the benefits of enterprise social networks.
  • A large, organizational survey to gain a robust, overall understanding of staff network experiences.
  • In-depth interviews with select user communities to gain a granular understanding of experiences.
  • Use of other large, highly-credible studies performed by academics and trade groups.  The findings of these studies are used to infer network benefits based on the experiences of many user communities in a variety of other organizations.
 
By combining the three approaches, this methodology balances its aims of (1) collecting credible, HAB-specific data; (2) making effective use of relevant, non-HAB specific, industry data; and (3) limiting staff burden caused by requests for information.
Estimating Network Engagement
Network engagement is both a key benefit of an enterprise social network and a driver of productivity gains and savings.  Network engagement promotes active staff involvement.  Network engagement has been shown to boost the productivity gains from the deployment of an enterprise social network.  The surest method of measuring network engagement is by recording observed behavior in the forms of management deployment and staff use of SP 2.0 applications.
Estimating Productivity Gains

A key industry study used in this methodology developed detailed estimates of productivity gains achieved by deploying enterprise social networks in many different office settings.  The productivity gains reflect time saved through greater ease in performing each of 11 basic job tasks.  A productivity gain may be inferred for each basic job task impacted by a given SP 2.0 application.
Estimating “Before” Profile

The TISTA SharePoint assessment presents a thorough review of HAB’s SharePoint network.  This methodology will draw from the TISTA study findings to develop a baseline or “Before” profile of HAB’s SharePoint work environment.  Any productivity gains attributable to initial SharePoint standup will be subtracted from estimates of SP 2.0 productivity gains to obtain a more accurate understanding of the incremental benefits due solely to the SharePoint improvement investments.

Estimating User Experience and Employee Empowerment Benefits
Research shows that the benefits of an enterprise social network stem from the affording of greater ease in performing basic job tasks.  

An enterprise social network can empower employees by affording greater independence of thought and action—for both individuals and teams.  For purposes of this methodology, the surest method of capturing such an improvement is to ask staff users of the network to compare their experiences before and after the new SharePoint structure and functionality.
Spotlight Analyses
The purpose of conducting 3 – 4 “Spotlight” Analyses is threefold: (1) to anchor the benefits analysis using selected anecdotes composed of familiar details on work tasks, processes, and conditions; (2) to provide a second estimate of productivity gains using an alternative method; and (3) to judge the overall “goodness of fit” of productivity gains estimates derived using more general, industry-level information.

4. Estimate ROI: Calculating ROI Output Measures
Rough estimates will be developed for all key benefits criteria and cost measures.  These rough estimates will serve as a first approximation and will be reviewed individually to assess their quality.  Judgments will then be made about whether gathering more data can substantially improve them.
 
Key ROI Analysis Output Measures
key-roi-analysys-output-measurements.jpg

5. Evaluate Uncertainties
The rough estimates of all ROI cost and benefit measures will be expressed in two ways: (1) an average (or some other measure of central tendency, as appropriate) or (2) a variance, representing the “spread” of individually-collected estimates around the average.  This spread is commonly used to gauge estimate uncertainty.  A high level of uncertainty can cast doubt on the representativeness of a cost or benefit estimate.  Uncertainty can be reduced, as necessary, by collecting additional data.
 
6. Refine Estimates to Increase Information Value and Reduce Uncertainty
Decisions about the need for additional data collection are made based on the information value of additional data.  ROI analysis findings may be highly sensitive to uncertainty.  By identifying the underlying source of uncertainty, it should be possible to do some limited, strategic data collection to strengthen ROI findings.
 
7. Finalize ROI Estimates for Project and Extensions
The final ROI findings for HAB’s SP 2.0 enterprise social network will include estimates of all the cost and benefit measures described previously.  The ROI findings will also include scaling factors to approximate the benefits that may be expected from additional network investments.
 
Scaling Factors

scaling-factors.jpg

These scaling factors will afford HAB decision makers the ability to ask “WHAT IF” questions about the benefits of new investments.  These new investments may be defined generally by dollar size or aimed at specific improvements—or functionality supporting job tasks (e.g., collaboration).

Strategic Alignment info...

strategic-alignment-info.jpg

Performance Measurement Methodology

  • Describe categories (definitions below)

  • Explain scoring of projects against key measures that fall under the main categories

  • Weigh the categories according to their priority for HAB


Performance Categories & Definitions

performance-categories-definitions.jpg
hab-performance-heirarchy.jpg
What Was Accomplished
  • Performance Hierarchy highlighting HAB organizational priorities through the lens of IT

performance-heirarchy-2.jpg
  • ROI results highlights

    • Organizational Insights

      • When HAB employees are equipped with the tools that best serve their needs, they are most successful in supporting the organization’s priority of effective service to external customers

      • It is critical for key leaders across HAB to clearly define the Bureau’s strategic priorities—and to coordinate on efforts to achieve the targets they set together

      • HAB has made significant strides in the last decade to stand up tools that support recipients and staff

      • Harnessing existing IT systems and tools to their fullest potential could yield significant benefits for HAB recipient and staff

      • Boosting capabilities for HAB staff could ultimately lead to better patient outcomes (in such areas as training, communication, and practice transformation)

  • Key Recommendations

    • Establish program governance for HAB IT investment management

    • Pursue tools that support cross-collaboration with additional partner agencies

    • Establish SMART goals to guide program direction and garner stakeholder support (for AETC and other programs)

    • Investigate feasibility and desirability of developing and/or expanding interoperability tools (for CAREWare and beyond)

    • Explore methods of providing faster access to client level data

    • Develop and implement targeted communication and training strategy to support HAB data user groups

    • Explore feasibility and desirability of modernizing RSR and ADR tools

    • Engage HAB stakeholders to align on future goals for data use and reporting (establish data vision and develop actionable strategy)

  • Major Successes

    • Timely and extensive ROI analyses, including an organization-wide survey deployment

    • Insightful feedback provided from the organization through engagement & cooperation with SMEs, key stakeholders, and HAB leaders

    • Standup of standard method to evaluate how well efforts align to organizational priorities

    • The outputs from our ROI efforts inform decisions both for (1) the evaluated projects and for (2) any other investments that HAB considers

  • The multi-year effort yielded the following overarching results:

    • Reusable ROI methodology for evaluating project value—embodying a holistic representation of both tangible and intangible costs and benefits of the specified HAB investments

    • A living hierarchy of performance measures that reflect organizational priorities—and that would be revisited and adjusted to remain in alignment with the evolving strategic direction of HAB

    • Foundation for tying performance to HAB priorities, TBM (Technology Business Management), FITARA (Federal IT Acquisition Reform Act), and the PMA CAP goals —including a scoring system to understand the comparative value of different efforts

    • Actionable recommendations: As of July 2021 , Innovate developed a total of 24 major recommendations based on its ROI study findings; these were crafted to improve HAB’s IT program effectiveness and efficiency, and promote better organizational health and innovation

Impact to HAB
  • Overarching value to stakeholders in support of the HAB mission

    • Across HAB

    • Within HRSA

    • Collaboration with partner organizations

    • Helping Research and Advocacy Groups

    • Serving Grant Recipients, their clinics, and their patients

  • What is means for HAB

    • Clearer line of sight from technology to mission execution

    • Equipping users with the tools they need to succeed in their roles

    • Better communication and collaboration, both within and outside of HAB

    • Better understanding of stakeholder needs

    • Better responsiveness to HAB employee needs--and higher potential for increased employee engagement and morale

    • Opportunity for closer relationships with recipients, increased understanding of clinical needs, and ultimately better patient outcomes

    • Using the right tools to impact public health -- both domestically and globally

What Happened Next
  • Steps HAB has taken in response to ROI results (findings & recommendations)

    • AETC follow up on recommendation: Establish SMART goals to guide program direction and garner stakeholder support

      • OPS has made progress incorporating strong federal leadership to this program.

      • Current efforts are underway to set priorities and define a path forward for AETC that aligns to HAB’s strategic direction

    • CAREWare follow up underway to explore recommendations

      • Cooperative Agreement feasibility to expand support options

      • Exploration of consolidated service options to reduce cost to HAB and users

      • Standup of a recipient Community of Practice to focus on importing data from EPIC to CAREWare (tied to need for interoperability between systems)

    • Data visualization efforts are underway to provide HAB project officers and State-level grant recipients with an enhanced user experience and easier interpretation of available data

  • Current effort and examples of focus areas

    • Holistic IT Portfolio Management for all HAB IT initiatives (incorporating HAB priorities and facilitating follow-up activity from completed ROIs)

    • Advanced Analysis work to focus on specific areas and define way forward (shift from observation to action-driven purpose of analyses; expansion of scope beyond ROI)

    • Automation and Collaboration efforts to build out functionality in the short term while sustaining collaborative environments for smooth user experience

Acknowledgements
  • Anna Kelner, Project Manager, Innovate!

  • Art Koines, Subject Matter Expert, Innovate!

  • John Tittle, Subject Matter Expert, Innovate!

  • Matt Brumberger, Subject Matter Expert, Innovate!

  • Phill Thomas, Subject Matter Expert, Innovate!