Designing metadata enrichments

Initial data scope

The Data assets list shows all assets of the supported formats. You can enrich relational and structured data assets. You can select individual assets, but you can also select metadata import assets to enrich the entire set of data assets from those metadata imports. However, you can't select data assets or metadata imports that are already included in a metadata enrichment. For individual data assets, you can hover over the asset name to see in which metadata enrichment the asset is included.

A metadata import asset is automatically excluded from the selection scope in these cases:

• It has a catalog as the import target.

• It was run on a connection that doesn't support access to the actual data.

  See Importing metadata.

  Remember: Each data asset or metadata import can be included in only one metadata enrichment per project. If you want to enrich a data asset several times with different enrichment options, you need to do that in separate projects.

If any of the connections for the selected data assets is configured to use personal instead of shared credentials, you must unlock that connection before you can proceed.

You can also create an empty metadata enrichment asset and set the scope later.

Scope of reruns of the enrichment

For reruns of the enrichment, whether scheduled or run manually, the data scope can be all assets from the initially selected data scope or a subset of assets. The default option is New and modified assets and assets not enriched in the previous run. With this option, assets are selected for enrichment as follows:

• Assets that were added after the last run of the enrichment
• Assets where columns were added or removed after the last run of the enrichment
• Assets where asset or column descriptions changed after the last run of the enrichment
• Assets for which the previous enrichment failed or was canceled

Enrichment is always run on the entire data asset regardless of whether an asset is new or modified.

The job run log shows reruns of metadata enrichments that are configured with the limited data scope as delta metadata enrichment job runs.

Profile data

Generate basic statistics about the asset content, assigns and suggests data classes.

This type of profiling is fast but makes some approximations for certain metrics like frequency distribution and uniqueness. To get more exact results without approximation, run advanced profiling on selected data assets. See Advanced data profiling. For more information about the statistics, see Detailed profiling results.

Data classes describe the contents of the data in the column: for example, city, account number, or credit card number. Data classes can be used to mask data with data protection rules or to restrict access to data assets with policies. In addition, they can contribute to term assignments if a corresponding data class to term linkage exists.

The confidence of a data class is the percentage of nonnull values that match the data class. The confidence score for a data class to be assigned or suggested must at least equal the set threshold. See Data class assignment settings. If a threshold is set on a data class directly, this threshold takes precedence when data classes are assigned. It is not considered for suggestions. In addition to the confidence score, the priority of a data class is taken into account.

Several data classes are more generic identifiers that are detected and assigned at a column level. These data classes are assigned when a more specific data class could not be identified at a value level. Generic identifiers always have a confidence of 100% and include the following data classes: code, identifier, indicator, quantity, and text.

Single-column primary keys are suggested based on profiling statistics. If primary key and foreign key constraints are already defined in your data and this information is included in the metadata import, these keys are automatically assigned.

From the enrichment results, you can run a multi-column primary key analysis where the actual data is checked. For more information, see Identifying primary keys.

Expand metadata

Generate semantic names and descriptions for data assets and columns. The names that exist in the source are expanded based on the collected metadata and a predefined glossary by using fuzzy matching and by comparing the names to business term abbreviation in the categories selected for the enrichment. If the asset or column name in the source can be matched to a business term abbreviation, the corresponding business term is used as the display name. Generative AI is used to provide descriptions based on the expanded names, surrounding columns, and the context of the data assets. Use this option to provide alternative names that are easier to consume than the often very technical original names. AI-generated descriptions can help to understand the content especially when column or data asset descriptions are missing in the data source. The assignment and suggestion thresholds are defined in the default enrichment settings.

Assign terms and classifications

Automatically assign business terms to columns and entire assets, or suggest business terms for manual assignment. Those assignments or suggestions are generated by a set of services. See Automatic term assignment.

Depending on which term assignment services are active for your project, term assignment might require profiling.

In addition, assign classifications to data assets and columns based on automatically assigned terms and data classes. Classification assignment must be enabled in the default enrichment settings. Classification assignment based on data classes also requires profiling.

Run basic quality analysis

Run predefined data quality checks on the columns of a data asset. The set of checks that is applied is defined in the enrichment settings. See Basic quality analysis settings. Each check can contribute to the asset's overall data quality cores. This type of data quality analysis can be done only in combination with profiling. Therefore, the Profile data option is automatically selected when you select to analyze data quality.

You can choose whether you want to write the output of these checks to a database. If default settings exist, the sections are populated accordingly. You can overwrite the settings. If no default settings exist, configure the output and the output location. For information about which data sources are supported as output target, see column Output tables in Supported data sources. Schema and table names must follow this convention:

• The first character for the name must be an alphabetic character.
• The rest of the name can consist of alphabetic characters, numeric characters, or underscores.
• The name must not contain spaces.

If you select to write the exceptions or the rows in which the issues were found (exception records) to existing tables, make sure these tables have the required format. See Data quality output.

If the connection that you pick is locked, you are asked to enter your personal credentials. This is a one-time step that permanently unlocks the connection for you.

Set relationships

Uses profiling statistics and name similarities between columns to provide primary and foreign keys and to suggest or assign relationships between assets and columns. The default enrichment settings for key relationships are applied. This type of relationship analysis requires profiling.

Run definition

Define when the metadata enrichment is run. You can select none, one, or both of these options:

Run after job creation

Select this option to run the metadata enrichment when you save a newly created metadata enrichment. Otherwise, the metadata enrichment asset is saved, but no job run is initiated.

Run on a schedule

Select this option to run the enrichment on a schedule. You can schedule single and recurring runs. Define the start date and time for the schedule. If you schedule a single run, the job runs exactly one time at the specified day and time.

To schedule recurring runs, select Repeat the job and the frequency in which you want the enrichment job to run. If you select Minutely, Hourly, or Daily, you can exclude certain days of the week from the schedule. Optionally, you can set an end date and time for the job schedule. For recurring runs, the job runs for the first time at the timestamp that is calculated based on the settings in the Repeat the job section.

Regardless of the run definition, you can manually trigger a run of the metadata enrichment job at any time.