Data Management Maturity Assessment

maturity assessment

data government

data management

Introduction

Capability Maturity Assessment (CMA) is an approach to process improvement based on a framework – a Capability Maturity Model (CMM) – that describes how characteristics of a process evolve from ad-hoc to optimal.

Maturity models are defined in terms of a progression through levels that describe process characteristics. Measure improvement and compare itself to competitors or partners, guided by the levels of the model.

Levels commonly include:

• Level 0: Absence of capability
• Level 1: Initial or Ad Hoc: Success depends on the competence of individuals
• Level 2: Repeatable: Minimum process discipline is in place
• Level 3: Defined: Standards are set and used
• Level 4: Managed: Processes are quantified and controlled
• Level 5: Optimized: Process improvement goals are quantified

Based on the findings, the organization can develop a roadmap to target:

• High-value improvement opportunities related to processes, methods, resources, and automation
• Capabilities that align with business strategy
• Governance processes for periodic evaluation of organizational progress based on characteristics in the model

A Data Management Maturity Assessment (DMMA) can be used to evaluate data management overall, or it can be used to focus on a single Knowledge Area or even a single process.

A DMMA can help bridge the gap between business and IT perspectives on the health and effectiveness of data management practices. A DMMA provides a common language for depicting what progress looks like across Data Management Knowledge Areas and offers a stage-based path to improvement, which can be tailored to an organization’s strategic priorities.

Thus, it can be used both to set and to measure organizational goals, as well as to compare one’s organization against other organizations or industry benchmarks.

Before beginning any DMMA, an organization has to establish a baseline understanding of its current state capabilities, assets, goals, and priorities.

---

Business Drivers

Organizations conduct capability maturity assessments for a number of reasons:

• Regulation: Regulatory oversight requires minimum levels of maturity in data management.
• Data Governance: The data governance function requires a maturity assessment for planning and compliance purposes.
• Organizational readiness for process improvement: An organization recognizes a need to improve its practices and begins by assessing its current state. For example, it makes a commitment to manage Master Data and needs to assess its readiness to deploy MDM processes and tools.
• Organizational change: An organizational change, such as a merger, presents data management challenges. A DMMA provides input for planning to meet these challenges.
• New technology: Advancements in technology offers new ways to manage and use data. The organization wants to understand the likelihood of successful adoption.
• Data management issues: There is need to address data quality issues or other data management challenges and the organization wants to baseline its current state in order to make better decisions about how to implement change.

---

Context Diagram

---

Goals and Principles

To evaluate the current state of critical data management activities in order to plan for improvement

Clarifying specific strengths and weaknesses

It helps the organization identify, prioritize, and implement improvement opportunities

DMMA helps:

• Educate stakeholders about data management concepts, principles, and practices.
• Clarify stakeholder roles and responsibilities in relation to organizational data.
• Highlight the need to manage data as a critical asset.
• Broaden recognition of data management activities across the organization.
• Contribute to improving the collaboration necessary for effective data governance.

Based on assessment results, an organization can enhance its Data Management program so it supports the organization’s operational and strategic direction.

A DMMA can equip the organization to develop a cohesive vision that supports overall organizational strategy. A DMMA enables the organization to clarify priorities, crystalize objectives, and develop an integrated plan for improvement.

---

Essential Concepts

Assessment Levels

The following is a generic summary of macro states of data management maturity.

• Level 0 - No Capability: No organized data management practices or formal enterprise processes for managing data.
• Level 1 - Initial / Ad Hoc: General-purpose data management using a limited tool set, with little or no governance.
  • Limited tool set.
  • Data handling is highly reliant on a few experts.
  • Roles and responsibilities are defined within silos.
  • Each data owner receives, generates, and sends data autonomously.
  • Controls, if they exist, are applied inconsistently.
  • Solutions for managing data are limited.
  • Data quality issues are pervasive but not addressed.
  • Infrastructure supports are at the business unit level.
• Level 2 - Repeatable: Emergence of consistent tools and role definition to support process execution.
  • Begins to use centralized tools
  • Provide more oversight for data management.
  • Roles are defined and processes are not dependent solely on specific experts.
  • There is organizational awareness of data quality issues and concepts.
  • Concepts of Master and Reference Data begin to be recognized.
• Level 3 - Defined: Emerging data management capability.
  • Introduction and institutionalization of scalable data management processes
  • View of DM as an organizational enabler.
  • The replication of data across an organization with some controls in place and a general increase in overall data quality, along with coordinated policy definition and management.
  • More formal process definition leads to a significant reduction in manual intervention.
  • A centralized design process, means that process outcomes are more predictable.
• Level 4 - Managed: Institutional knowledge gained from growth in Levels 1-3 enables the organization to predict results when approaching new projects and tasks and to begin to manage risks related to data.
  • Data management includes performance metrics.
  • Standardized tools for data management from desktop to infrastructure, coupled with a well-formed centralized planning and governance function.
  • Expressions of this level are a measurable increase in data quality and organization-wide capabilities such as end-to-end data audits.
• Level 5 - Optimization: When data management practices are optimized, they are highly predictable, due to process automation and technology change management.
  • Focus on continuous improvement.
  • Tools enable a view data across processes.
  • The proliferation of data is controlled to prevent needless duplication.
  • Well-understood metrics are used to manage and measure data quality and processes.

Assessment Criteria

• level 1 may ask whether a data modeling practice exists at all and how many systems it extends to.
• level 2 may ask whether an approach to enterprise data modeling has been defined.
• level 3 will ask the degree to which the approach has been implemented
• level 4 will ask whether modeling standards have been effectively enforced
• level 5 will ask about processes in place to improve modeling practices.

At any level, assessment criteria will be evaluated along a scale, such as:

• 1 – Not started
• 2 – In process
• 3 – Functional
• 4 – Effective showing progress within that level and movement toward the next level.

Criteria could be formulated based on the categories in the Context Diagram:

• Activity:
  • To what degree is the activity or process in place?
  • Are criteria defined for effective and efficient execution?
  • How well defined and executed is the activity? Are best practice outputs produced?
• Tools:
  • To what degree is the activity automated and supported by a common set of tools?
  • Is tool training provided within specific roles and responsibilities?
  • Are tools available when and where needed?
  • Are they configured optimally to provide the most effective and efficient results?
  • To what extent is long-term technology planning in place to accommodate future state capabilities?
• Standards:
  • To what degree is the activity supported by a common set of standards?
  • How well documented are the standards?
  • Are standards enforced and supported by governance and change management?
• People and resources:
  • To what degree is the organization staffed to carry out the activity?
  • What specific skills, training, and knowledge are necessary to execute the activity?
  • How well are roles and responsibilities defined?

We could create 2 values to identified level of capability.

• The higher one shows the level of capability the organization has determined it needs to compete successfully. The lower one shows the level of capability as determined via the assessment.

Areas where the distance between the two values is largest represent the greatest risks to the organization. Such a report can help set priorities. It can also be used to measure progress over time.

---

Models

• CMMI (Capability Maturity Model Institute) DMM (Data Management Maturity Model). Assessment criteria for the following data management areas:
  • Data Management Strategy
  • Data Governance
  • Data Quality
  • Platform and Architecture
  • Data Operations
  • Supporting Processes

  Within each of these processes, the model identifies sub-processes for evaluation. The model also accounts for the relation between the data management areas.

• EDM (The Enterprise Data Management) Council DCAM (Data Management Capability Assessment Model). the DCAM describes 37 capabilities and 115 sub-capabilities associated with the development of a sustainable Data Management program. Scoring focuses on the level of stakeholder engagement, formality of process, and existence of artifacts that demonstrate the achievement of capabilities.

• IBM Data Governance Council Maturity Model The purpose of the model is to help organizations build consistency and quality control in governance through proven business technologies, collaborative methods, and best practices. The model is organized around four key categories:
  • Outcomes: Data risk management and compliance, value creation.
  • Enablers: Organizational structure and awareness, policy, stewardship.
  • Core disciplines: Data Quality Management, information lifecycle management, information security and privacy.
  • Supporting Disciplines: Data Architecture, classification and Metadata, audit information, logging and reporting.

• Stanford Data Governance Maturity Model The model focuses on data governance, not data management. The model differentiates between foundational (awareness, formalization, Metadata) and project (data stewardship, Data Quality, Master Data) components. Within each, it articulates drivers for people, policies, and capabilities. It then articulates characteristics of each level of maturity. It also provides qualitative and quantitative measurements for each level.

• Gartner’s Enterprise Information Management Maturity Model Gartner has published an EIM maturity model, which establishes criteria for evaluating vision, strategy, metrics, governance, roles and responsibilities, lifecycle, and infrastructure.

---

Activities

Assessments should be conducted in a short, defined timeframe.

The purpose of the evaluation is expose current strengths and opportunities for improvement – not to solve problems.

Evaluations are conducted by soliciting knowledge from business, data management, and information technology participants. The goal is to reach a consensus view of current state capabilities, supported by evidence. Evidence may come from examination of artifacts, interviews, or both.

Assessments can and should be scaled to fit the needs of the organization. However, amend with care. Models may lose rigor or traceability to original intent if shortened or edited. Keep the integrity of the model intact when customizing.

---

Plan Assessment Activities

Defining the overall approach and communicating with stakeholders before and during the assessment to ensure they are engaged. The assessment itself includes collecting and evaluating inputs and communicating results, recommendations, and action plans.

Define Objectives

Any organization that decides it should assess its data management maturity level is already engaged in the effort to improve its practices.

The objectives for the assessment must be clearly understood by executives and the lines of business, who can help ensure alignment with the organization’s strategic direction.

• Assessment objectives also provide criteria by which to evaluate which assessment model to adopt,
• Which business areas to prioritize for assessment
• Who should provide direct input to the process.

Choose a Framework

Choose one that will inform the organization in meaningful ways. Focus areas of the assessment model can be customized based on organizational focus or scope.

Define Organizational Scope

For a first assessment, it is usually best to define a manageable scope, such as a single business area or program.

The areas chosen represent a meaningful subset of the organization and participants should be able to influence key business processes that affect the data assets within scope.

There are trade-offs between local and enterprise assessments:

• Localized assessments:
  • Can go much deeper into the details.
  • Done more quickly because the scope is contained.
  • Select a function that is highly regulated, such as financial reporting within a public company.
  • Can often be aggregated and weighted to form an enterprise assessment, since many data assets are shared.
• Enterprise assessments:
  • Focus on the broad and sometimes disconnected parts of an organization.
  • For example, an organization may evaluate different functions (research and development, manufacturing, and financing) based on the same criteria.

Define Interaction Approach

Information gathering activities may include:

• workshops
• interviews
• surveys
• artifact reviews

Employ methods that work well within the organizational culture, minimize the time commitment from participants, and enable the assessment to be completed quickly so that actions from the assessment can be defined while the process is fresh in participants’ minds.

In all cases, responses will need to be formalized by having participants rate the assessment criteria. In many cases, assessment will also include actual inspection and evaluation of artifacts and other evidence.

If there are delays in completing the assessment, stakeholders are likely to lose enthusiasm for the Data Management program and the impetus for contributing to positive change.

It is advisable to avoid detailed and comprehensive analysis and to emphasize sound judgment based on the expertise of the assessment leaders.

Plan Communications

Findings may impact people’s jobs, through changes in methodology and organizational alignment, so it is important to communicate clearly about the purpose, the process, and specific expectations for individuals and groups.

Stakeholders should be informed about expectations for the assessment. Communications should describe:

• The purpose of the DMMA
• How it will be conducted
• What their involvement may be
• The schedule of assessment activities

Continually remind participants of the goals and objectives. Always thank the participants and describe next steps.

Determine if the planned approach is likely to be successful across the targeted business scope, including such factors as resistance / cooperation, possible internal legal concerns about exposure to outside inspection if troubling gaps are found, or possible Human Resources concerns.

The communications plan should include a schedule to report on findings and recommendations at all levels, including general reports and executive briefings.

---

Perform Maturity Assessment

Gather Information

At a minimum, the information gathered will include formal ratings of assessment criteria. It may also include input from interviews and focus groups, system analysis and design documentation, data investigation, email strings, procedure manuals, standards, policies, file repositories, approval workflows, various work products, Metadata repositories, data and integration reference architectures, templates, and forms.

Perform the Assessment

Input is provided by the participants and then refined through artifact reviews or examination by the assessment team. The goal is to come to a consensus view of current state.

This view should be supported by evidence (i.e., proof of practice demonstrated by behavior and artifacts). If stakeholders do not have consensus on current state, it is difficult to have consensus on how to improve the organization.

The refinement generally works as follows:

• Review results against the rating method and assign a preliminary rating to each work product or activity.
• Document the supporting evidence.
• Review with participants to come to consensus on a final rating for each area. If appropriate, use weight modifiers based on the importance of each criterion.
• Document the interpretation of the rating using the model criteria statements and assessor comments.
• Develop visualizations to illustrate results of the assessment.

---

Interpret Results

Identifying improvement opportunities aligned with organizational strategy and recommending actions required to take advantage of these opportunities. Interpretation defines next steps toward a target state.

When the assessment is complete, organizations need to plan for the target state that they aspire to achieve in data management. The amount of time and effort required to achieve the desired target will vary, depending on the starting point, the culture of the organization, and the drivers for change.

When presenting assessment results, start with the meaning of the ratings for the organization. The ratings can be expressed with respect to organizational and cultural drivers as well as business goals, such as customer satisfaction or increased sales. Illustrate the linkage between the current capabilities of the organization and the business processes and strategies that they support, and the benefits of improving these capabilities by moving to the target state.

Report Assessment Results

The assessment report should include:

• Business drivers for the assessment
• Overall results of the assessment
• Ratings by topic with gaps indicated
• A recommended approach to close gaps
• Strengths of the organization as observed
• Risks to progress
• Investment and outcomes options
• Governance and metrics to measure progress
• Resource analysis and potential future utilization
• Artifacts that can be used or re-used within the organization

Strategy should include initiatives that further business goals through improved governance of processes and standards.

Develop Executive Briefings

This is a summarize findings – strengths, gaps, and recommendations – that executives will use as input to decisions about targets, initiatives, and timelines. The team must tailor the messages to clarify likely impacts and benefits for each executive group.

Targeting a higher level of maturity has to be reflected in the impact analysis for the recommendations. There is a cost to this kind of acceleration, and costs must be balanced against benefits.

---

Create a Targeted Program for Improvements

Recommendations from the DMMA should be actionable

Identify Actions and Create a Roadmap

DMMA ratings highlight items for management attention. Ratings can be quickly operationalized into ongoing measures, especially for activities where change is desired (e.g., “The target is level ‘n’ because we need or want to be able to do something ‘z’”).

If the assessment model is used for ongoing measurement, its criteria not only guides the organization to higher levels of maturity, its criteria also keeps organizational attention on improvement efforts.

The DMM assessment results should support a multiple year data management improvement program, including initiatives that will build data management capability as the organization adopt best practices.

The roadmap or reference plan should contain:

• Sequenced activities to effect improvements in specific data management functions
• A timeline for implementing improvement activities
• Expected improvements in DMMA ratings once activities have been implemented
• Oversight activities, including the maturing this oversight over the timeline

---

Reassess Maturity

They are part of the cycle of continuous improvement:

• Establish a baseline rating through the first assessment
• Define reassessment parameters, including organizational scope
• Repeat DMM assessment as necessary on a published schedule
• Track trends relative to the initial baseline
• Develop recommendations based on the reassessment findings

---

Tools

• Data Management Maturity Framework
• Communication Plan
• Collaboration Tools
• Knowledge Management and Metadata Repositories Data standards, policies, methods, agendas, minutes of meetings or decisions, and business and technical artifacts that serve as proof of practice may be managed in these repositories. In some CMMs, lack of such repositories is an indicator of lower maturity in the organization.

---

Selecting a DMM Framework

The following criteria should be considered when selecting a DMM framework.

• Accessibility: Stated in non-technical terms.
• Comprehensiveness: Addresses a broad scope of data management activities.
• Extensible and flexible: Enable enhancement of industry-specific or additional disciplines in whole or in part.
• Future progress path built-in: Outlines a logical way forward within each of the functions it describes.
• Industry-agnostic vs. industry-specific: Adhere to data management best practices that cross verticals.
• Level of abstraction or detail: Practices and evaluation criteria are expressed at a sufficient level of detail.
• Non-prescriptive: The framework describes what needs to be performed, not how it must be performed.
• Organized by topic: Places data management activities in their appropriate context, enabling each to be evaluated separately, while recognizing the dependencies.
• Repeatable: The framework can be consistently interpreted, supporting repeatable results to compare an organization against others in its industry and to track progress over time.
• Supported by a neutral, independent organization: The model should be vendor neutral in order to avoid conflicts of interest, and widely available to ensure a broad representation of best practices.
• Technology neutral: The focus of the model should be on practices, rather than tools.
• Training support included: The model is supported by comprehensive training to enable professionals to master the framework and optimize its use.

---

Readiness Assessment / Risk Assessment

it is helpful to identify potential risks and some risk mitigation strategies.

---

Organizational and Cultural Change

Establishing or enhancing a Data Management program includes changes to processes, methods, and tools. With these changes, culture must also change. Organizational and cultural transformation begins with acknowledging that things can be better.

The process should have an executive sponsor, to ensure improvements in data management activities map directly to business objectives.

---

Metrics

Initial DMMA metrics are the ratings representing the current state of data management.

Sample metrics could include:

• DMMA ratings: Present a snapshot of the organization’s capability level. perhaps a custom weighting for the rating across an assessment or specific topic area, and a recommended target state.
• Resource utilization rates: Powerful examples of metrics that help express the cost of data management in the form of head count. An example of this type of metric is: “Every resource in the organization spends 10% of their time manually aggregating data.”
• Risk exposure or the ability to respond to risk scenarios expresses an organization’s capabilities relative to their DMMA ratings. For example, if an organization wanted to begin a new business that required a high level of automation but their current operating model is based on manual data management (Level 1), they would be at risk of not delivering.
• Spend management expresses how the cost of data management is allocated across an organization and identifies the impacts of this cost on sustainability and value. These metrics overlap with data governance metrics.
• Inputs to the DMMA are important to manage as they speak to the completeness of coverage, level of investigation, and detail of the scope relevant for interpretation of the scoring results. Core inputs could include the following: count, coverage, availability, number of systems, data volumes, teams involved, etc.
• Rate of Change The rate at which an organization is improving its capability. A baseline is established through the DMMA. Periodic reassessment is used to trend improvement.