What is a data catalogue, and do you need one?

Back in 2019, Gartner reported: “Demand for data catalogues is soaring as organisations continue to struggle with finding, inventorying and analysing vastly distributed and diverse data assets." Unfortunately, this year’s Data Health Barometer shows that many still struggle to curate their datasets effectively. Big data is only getting bigger, but organisations still face metadata management and cataloguing challenges.

A data catalogue could become the cornerstone of your data-driven strategy.
This page will help you get familiar with the enterprise data catalogue and its benefits across the organisation. Learn what a data catalogue is, and how a data catalogue can help you build a data culture.

What is an enterprise data catalogue?

Gartner defines the data catalogue category in another report. “A data catalogue maintains an inventory of data assets through the discovery, description, and organisation of datasets. The catalogue provides context to enable data analysts, data scientists, data stewards, and other data consumers to find and understand a relevant dataset for the purpose of extracting business value.”

This description is a good start, but it might be too restrictive. True, the most visible data catalogue use case is self-service data discovery. But a complete data catalogue solution doesn’t just help end users to find and understand data. Behind the scenes, it also automates your metadata management and governance processes.

What is metadata management and why is it important?

Metadata is data about data. ISO 15489 defines metadata for records as: “structured or semi-structured information, which enables the creation, management, and use of records through time and within and across domains.” Think of names, creation dates, and any other contextual information that describes the data in your data lake or data warehouse. All this metadata adds meaningful information to your datasets. This improves the data’s usability and makes data a real asset for your organisation. A catalogue of all the metadata makes search and retrieval of any data possible.

To illustrate, think of an online catalogue for finding books in a library. The catalogue is a centrally managed source of truth where readers can look up any given book. The catalogue curates metadata for each book including title, author, summary, publication date, and shelf location. It might also include other readers’ ratings and recommendations, and a way to leave your own reviews. The online catalogue lets library patrons find and access what they need faster than if they walked through the library shelf by shelf. Of course, all the metadata has to be kept well-organised and kept up-to-date in order for users to trust it and use it.

Metadata management means tracking and curating data assets across the complete data lifecycle. Active processes ensure consistency as flows through the system and you add new data sources. Poor metadata management results in siloed data, data quality problems, and . The inability to find data easily means data can’t deliver on the value. Worse, inability to find and account for data can make it impossible to comply with data regulations. In short, good metadata management is critical to delivering trusted data.

Different types of users need access to different types of metadata. For a business user, it may be enough to see data provenance, for example where a record originated. However, the IT team deploying ETL processes needs more technical metadata. Full-fledged data lineage capabilities provide IT users with a detailed itinerary of each record’s route from source to endpoint. They’ll see not only where data came from, but everywhere it’s been and what kind of processing occurred at every step. That data lineage gives IT users data visibility so they can easily identify and fix breaks in their data pipelines.

Why do we need metadata management?

Your data catalogue will become the single source of trust to unify all the metadata across your organisation. It can automatically discover, profile, organise and document your metadata, and makes it easily searchable. But who needs metadata management, and why? It turns out that there are three data catalogue use cases, with specific benefits for different audiences. Watch this video about the three audiences of a data catalogue:

Metadata management for IT users

A data catalogue tool reduces workloads for IT users tasked with metadata management. As we see across industries, robust automation and machine learning features remove busy work, freeing up human effort to do the critical work that only humans can do.

Data governance for data policy and compliance managers

Data risk management is increasingly vital in today’s increasingly regulated data landscape. Fortunately, data governance users can implement data governance policies within the data catalogue, tagging sensitive data and automating data governance rules.

Building governance into your data catalogue maximises data availability and data privacy. Data becomes more discoverable and accessible to those who need it, while privacy management is built into workflows. Some data catalogues have a one-size-fits-all approach that may not fit every situation. Talend Data Catalog enables flexible metamodels. You can tailor your data catalogue to your unique data landscape and governance needs.

Data discovery and context for data analysts and business users

A full-featured data catalogue tool gives data consumers the means to find, understand, and contribute to datasets relevant to their roles. When a data catalogue is well-organised, with diligent metadata management and built-in data governance rules, self-service data access maximises data usage while limiting data risk.

What are the benefits of a data catalogue?

A collaborative data catalogue tool like Talend Data Catalog makes data management a team sport for end users to participate in. Collaborative metadata management features tap into the subject matter expertise of business users. When data users review and comment on the data they work with, they add to the so-called “tribal knowledge” of the organisation.

As discussed above, there are three primary audiences for the data catalogue. Each of them derives a benefit from the data catalogue tool that positively impacts your bottom line:

• Reducing losses caused by downtime — automating metadata management helps IT teams monitor for broken data pipelines as schemas change, and collaborate to quickly repair them
• Reducing risk of noncompliance fines — building data governance into your data catalogue ensures adherence to any relevant data policies or data regulations
• Data-driven decisions that increase revenue and cut costs — data is only valuable if it’s actionable, and a data catalogue helps deliver data that makes data-driven decisionmaking possible

Perhaps the biggest benefit of a data catalogue is its role in supporting your organization’s data culture. You can’t have data health without an effective data culture. The collaborative features mentioned above put business and tech teams on the same page, sharing a single source of truth across the business.

Business glossaries within the data catalogue also establish shared definitions of business terms relating to data. The business glossary ensures that business users, data analysts, and IT are all literally speaking the same language about data. This data catalogue feature helps everyone understand, find, and track data with certain qualities. That shared data literacy across teams and functions is the foundation for a unified data culture.

What does a “modern” data catalogue mean?

Now that you have an introduction to the concept of a data catalogue, it’s time to ask the question, “what is a modern data catalogue?” Let’s look at another analogy we can all relate to: the Amazon marketplace.

Amazon is linked to all kinds of shops, retailers, and e-tailers. As a customer, you experience it all as one seamless shopping experience. A data catalogue can be just as powerful and useful an experience for data consumers. It’s the Amazon of all your data. And like on Amazon, someone is managing that data behind the scenes with the power to curate all the data. Data owners, like shop owners on the Amazon marketplace, are equipped with tools to curate and cleanse their own data. End users can see where data comes from, see notes from other users, and gain trust in what’s available to them. Like an Amazon shopping experience, your catalogue can become a living marketplace for all the valued data within your company.

Curating all that information manually is a challenging and time-consuming operation. As big data truly gets big with higher volumes of data coming in from real-time data streams, it’s becoming impossible to manage without the help of automation and artificial intelligence. Fortunately, modern data catalogues have an extensive range of powerful capabilities such as pattern detection, relationship discovery, pervasive profiling, automatic harvesting and classification so you can highlight data quality issues very easily. Machine learning and automation mean you can start applying corrective actions without constant manual effort.

Key ingredients of a successful data catalogue

Not all enterprise data catalogues are created equal. When choosing between data catalogue tools, it’s essential to filter players based on key capabilities. These are the key components that Talend Data Catalog offers to make your data strategy successful:

• Connectors and curation tools to build a single centre of trusted data. A data catalogue tool needs connectors to harvest metadata from wide-ranging sources. Having a wide array of connectors reinforces a data catalog’s ability to harvest any data and metadata, whatever the nature or the source of those datasets. Data sources will likely include business intelligence tools, data integration tools, SQL queries, enterprise apps like Salesforce or SAP, data modelling tools, and Internet of Things (IoT) devices delivering real-time metrics.
• Collaborative data curation capabilities. Building a single source of trust should rely not only on data source connecting capabilities but also on validation and certification tools that allow users to add metadata. With user scoring, comments, and reviews, your team can make your data governance strategy a living process over time.
• Automation to gain speed and agility. With enhanced automation, data stewards won’t spend time connecting data sources manually. They will then focus on what’s really important — correcting data quality issues and curating it for the benefit of the whole organisation. Of course, you will supplement automation with stewards’ help to enrich and curate datasets over time.
• Powerful search to easily explore datasets. Since data discovery is the primary function of a data catalogue for end users, search is the key to a good user experience. Search should be multi-faceted so you can specify different parameters to perform an advanced search. Search parameters including name, size, time, owner, and format are incredibly helpful for data discovery.
• Data lineage to perform root cause analysis. Data lineage helps you to link a dashboard to the data it exposes. Lineage and relationship discovery play a big role in understanding the relationship between different types and sources of data. So, if your dashboard displays inconsistent data, a steward can use the lineage to see where in the pipeline the problem is coming from. We can take the same approach to spot applications containing shadow IT that escape to IT’s control. That way, IT can better protect datasets using consumer databases with PII data.
• Glossary to crystallise business terms related to data. For everyone to use data consistently across the business, they need a common language. A business glossary creates a shared understanding of terms and definitions. By linking definitions to the data itself the business glossary becomes actionable. For example, search for PII in a data catalogue and you will find data sources containing personally identifying information. This is very useful in a GDPR context where you need to take control of all data with personal information.
• Profiling to stop polluting your data lake or data warehouse. When connecting different data sources, data profiling is key to assessing your data quality for completeness, accuracy, timeliness and consistency. Profiling saves time and helps you to quickly spot inaccuracies, so you can make stewards aware of this issue before polluting the data repository.

For people to become accountable within a data culture, you need to give them the right tools. It must be easy to discover, trust, and act on their data. You will realise your data catalog’s ultimate value by linking it with additional self-service tools. Imagine if data stewards and business users could prepare datasets and curate your data over time.

Learn more about data catalogues

A data catalogue tool is vital for metadata management, governance, and data discovery. If you need to take control of your data, build a trusted data culture the collaborative way, implement a data governance strategy to lock down workflows for privacy regulations such as GDPR, CCPA, and HIPAA, then you need a data catalogue. Explore Talend Data Catalog. It’s a secure, single point of control for your organisation’s data and metadata with connectors to extract metadata from virtually any source.