Data Discovery

Collate supports full-text search, AI-based natural language search, browsing, filtering, and discovery via relationships, i.e., through lineage graphs and through entity-relationship diagrams (ERDs).

First, Collate provides all the search capabilities you need to find or explore data, including an AI-based natural language search assistant. Combined with a rich metadata specification, the platform lets you find data that is relevant to your query without necessarily matching the exact terms (e.g., search for “clients” and also get “customers” data). Second, Collate provides a comprehensive data quality testing environment that lets you understand the health of your data. This lets you understand that the data you find is reliable and ready for use. Third, Collate provides querying and collaboration tools so you can get further validation on the usefulness of the data you find.

Collate data discovery is powerful because the platform captures all the information you need, along with relationships between metadata (in a “unified semantic graph”), to provide a complete understanding of your data. This makes discovery more effective, as the system lets you find information that might not be explicitly in your search terms, but still relevant. Collate also supports Search Relevancy Settings that let you configure the weighting of queries at a per-user level, allowing the system to return the data that is most relevant to that particular user.

Data discovery is the practice of looking for data you need via search, browse, and other exploration methods. It is typically only one part of a bigger data management strategy that also includes capabilities around data quality, data observability, lineage, and governance.

Data discovery is hard because data requires a lot of context for it to be effectively searchable by any user. Tagging is one way to add context so that search engines can find the data that users seek. But even more important is including data meaning in the equation. This is addressed in the Collate Semantic Intelligence Platform via a unified semantic graph, which captures relationships between terms. For example, you might have data tagged as “PII” but then have a semantic graph that links PII with GDPR. If a user searches for GDPR, they will get the data tagged with PII because the system saw the association between PII and GDPR. Note that the data didn’t have to be explicitly tagged with GDPR, which results in significant time savings from the effort of tagging all data sets that are tagged with PII. This is just one simple example of how a semantic graph can be useful in discovery. Some relationships might be very complex, but fortunately in Collate, you only need to define the relationship once in the system and all relevant assets will inherit those relationships. Another difficulty around discovery is that just because you found the data that matches your search criteria doesn’t mean the data is ready for use. You might find data that hasn’t been tested, or is not production ready. Without appropriate metadata, you might end up with a garbage-in-garbage-out situation. Data discovery capabilities necessarily need to incorporate data health and quality information, along with usage patterns to validate the data set is ready for production use. Collate provides all the information you need as part of its data discovery capabilities to ensure that the data you find is trusted, reliable, and ready for use.

Self-service data access is the practice of giving tools to any type of user so they can take advantage of data, as long as they’re given the appropriate permissions. This practice frees up data teams from searching for data on behalf of users, allowing them to focus on more strategic tasks.

Impact analysis is the practice of understanding how changes you make will affect other assets. This is an important practice that safeguards against the risk of breaking your data operations because of a change. This is an important aspect of data discovery because if you find data that you need to modify/transform, you want to make sure that your changes are safe and won’t cause a disaster.

The ability to find sensitive data for compliance depends on proper tagging of your data. This tagging often starts as a manual process, but automated techniques involving AI tagging, metadata propagation, and reverse metadata help to tag data across its journey. In a typical system, data needs to be tagged with all the right terms for it to be properly findable. For example, you might need to tag a data set with GDPR, CCPA, PCI, HIPAA, and DPDP. But with Collate, you can ask AI to tag sensitive data with PII and then create a semantic graph that connects PII to all of the privacy regulations. And then you can use metadata propagation to further label related data sets as PII. Now any user can search for PII and get all the data sets that are covered by the various regulations.

In Collate, all assets are first run through a “metadata ingestion” process that involves pointing the native connector to the data asset. Then a profiler is run to collect metrics. From there, all collected metadata is useful not only for discovery, but also as a starting point for data quality, lineage, and governance. In other words, the process for setting up assets for discovery also gets you most of the data you need to run other data management practices in Collate, so there’s no added complexity for doing more.