Link Search Menu Expand Document
Personal Projects

Impact Measurement

dashboard

data strategic

drawing

Main Idea

The problem you are solving (product/service) is aligned with your company’s objectives

How to measure the impactness of project

  • Is this new?
    • Create a questioner
      • Alignment with business and get business sponsorship
  • Is this improvement?
    • Create a measurement test
      • A/B testing
      • Experimentation
      • Evaluation

How relation between product/service and business metrics

  • Fundamental
  • Enabler
  • Direct

For fundamental and enabler relation, it requires large support for business sponsorship due to it opens new streamline. For direct relation, we need define comprehensive analysis and measurement, list all what we could control and we couldn’t, how it align with business metrics.

Core components for measuring impact P.U.G.E.T.

The launch of any new product of feature entails risk. It is important to understand the risk and determine related counter metrics to help track success

  • Product/problem definition
  • Users and customer segments
  • Goals and metrics
  • Efficient and measurable strategy
  • Trade-offs

How to quantify the impact

  • Divide data and product metrics into 3 values
    • Reach: The number of customers this problem is impacting. This is a real, objective number.
    • Intensity: How deep the pain caused by this problem is. This one is a slightly arbitrary scale of low-medium-high, or you can use a scale of 1–10, up to you.
    • User segment: Who are the customers impacted by this problem. Some of your customers are more valuable to the business than others.
  • Formula:

    Impact = reach * intensity * user segment

Main Impact Measurement

  • Monetery
    • Cost Reduction
    • Increase Revenue and profit
  • Production
    • Yield Improvement
    • Process Improvement
    • Better Quality
    • Human Time Saving
  • Time
    • Speed (SLA)
  • Customers
    • Market Share
    • More Customers
    • ROI

How a product / service data is develop

drawing

  • Proof of Concept (POC)
    • As an exploration to validate the feasibility of an idea.
    • Typically scoped down to just one segment of clients, one country, etc.
  • Prototype
    • Aims to create a functional representation of the final ML model.
    • Involve a larger dataset and a comprehensive
    • Implementation of the ML pipeline.
    • Iterative development and testing.
    • Provide feedback.
    • Refine the model’s design and functionality.
  • Minimum Viable Product (MVP)
    • Emphasizing the minimum set of features necessary to deliver value to users or stakeholders
    • A deployable ML model that provides a practical solution to a specific problem. focuses on delivering core functionality.
  • Production
  • Enterprise

drawing drawing

Objective level

To measure the impact of product/service we should breakdown the objective company from high to low objective.

  • Company Goal
  • Enterprise level
  • Business Function objective - and it relation with other division
  • Team objective
  • Individual objective

Workflow

  • Schedule a discovery sessions
  • Choose the right metrics
    • Define success criteria
  • Establish the baseline for each metrics
    • your numbers do not have to be accurate, rather a high-level approximation.
  • Wait for certain period after deploying your DS solution and then collect the new raw data for the same metrics.
    • Develop features
    • Dogfood
    • Conduct A/B testing
    • Research
    • Release for general availability
  • The Collection for the next learning
    • Making process repeatable and trackable. Can move from bespoke “artisan thinking” to reproducible and reusable “modular system” thinking.
  • Lunch-and-learns
    • Everyone learns what your team has done and where it’s going next.
    • Conducts monthly lunch presentations on projects, methodologies, and data usages.
    • The team also hosts an annual data science expo and regular forums featuring external guest speakers who educate and inspire.

Example questions in questioner

  1. Is this problem merely a symptom of a bigger problem?
    • 5-whys to find root cause
    • What is the purpose of the project?
    • How and by whom will it be used?
    • What problem are you trying to solve?
  2. Determine at what level is this problem?
  3. How impactful is this problem for the customers?
    • Reach → The number of customers
    • Intensity → How deep the pain caused by this problem is.
    • User segment → Who are the customers impacted by this problem.

      Impact = reach * intensity * user segment

  4. How will solving this problem benefit the company?
    • Short term
    • Long term
    • What value should the model bring to the company?
  5. Does it align with the company/product’s long-term vision and strategy?
  6. What do you need to de-prioritize to work on this?
  7. What happens if you do nothing?
    • a fire — you either have to solve them now or rebuild later
    • a leaking roof — it gets worse and worse slowly until the whole roof falls down.
    • a puddle of water — it’s annoying but it doesn’t harm anyone as long as everyone knows to walk around it.

  8. Imagine we’ve done all the hard work and developed a perfect solution. What will you do with it?

  9. Define success criteria or right matrices
    • new
    • optimization