Reviewing metadata enrichment results

Asset and enrichment details

Access asset and enrichment details by clicking the asset name or by clicking View asset details from the overflow menu. On the Details tab in the side panel, you find this information:

• The source of the data asset: the connection and database for connected assets. For files uploaded from a local system, Project is shown in the Source column.
• Asset details: the table type, the number of columns and rows in the asset, and its data format.
• The asset owner. The asset owner is usually the user who added the asset to the project except for assets that were added from a catalog. In this case, the catalog asset owner is also the project asset owner.
• The selected enrichment options.
• The sampling method.
• The date when the asset was last enriched and a link to the details of that enrichment job run.
• The asset description.

Governance information

Governance information for an asset includes assigned and suggested business terms and assigned classifications, which are listed in the Business terms and Classifications columns of the results. For assigned terms, a purple dot indicates that at least one of the terms was automatically assigned.

Access detailed governance information for an asset by clicking the asset name, by clicking the View more link in the Business terms or Classifications column, or by clicking View asset details from the overflow menu. On the Governance tab in the side panel, you can manage term and classification assignments.

Information about primary keys and relationships

Access key and relationship information for an asset by clicking the asset name, by clicking the View more link in the Primary key column, or by clicking View asset details from the overflow menu. On the Keys tab in the side panel, you can manage key assignments and relationships.

For primary keys that were identified through primary key analysis, the number and percentage of unique values, the number and percentage of null values, and the number of analyzed columns is shown. This information is not available for a key that was picked and assigned manually without prior primary key analysis, or for suggested primary keys that result from in-depth key relationship analysis.

The Relationships section provides these views of the assigned relationships:

• Parent of tab: In relationships to the listed assets, the current asset provides the primary key.
• Child of tab: In relationships to the listed assets, the current asset provides the foreign key.

If a relationship analysis was run but no relationships are assigned yet, you can click the plus icon to view and work with the analysis results.

At any time, you can edit the assigned relationships by clicking the pencil icon.

For more information, see Identifying primary keys and Identifying relationships.

Data quality score

A data quality score is displayed only if at least one data quality check was applied to the asset. Otherwise, a dash (—) is shown. The score shown for a data asset is the weighted average of the scores provided by the columns in the data asset. Data quality scores that are below the specified threshold are marked with a red dot. Data quality scores that are equal to or exceed the specified threshold are marked green.

A delta value shows how the overall data quality score changed compared to the score from 90 days before the latest analysis:

• A green arrow pointing to the upper right () indicates that the data quality score went up.
• A red arrow pointing to the lower right () indicates that the data quality score went down.

To find assets with quality issues quickly, especially when the enrichment scope is large, you can filter the list by quality scores.

For details about data quality issues, select an asset and click View data quality details from the overflow menu, or click the quality score.

For more information, see Data quality analysis results and Data quality scores.

Review status

Initially, the review status of all assets in the metadata enrichment is Not reviewed. After you review the enrichment results for an asset in the asset profile, you can set the asset's review status to Reviewed. Thus, everybody on the team is aware of what already was looked at and what still needs to be reviewed. If a later enrichment run updates the results of an asset with the status Reviewed, the asset's review status is set to Reanalyzed after review (). The review status does not change for updates to the asset that were found during metadata import.

Note that, for assets that are marked as reviewed, term assignments are not updated on reruns of an enrichment. For more information, see How new analysis results update existing term assignments.

Filter the list of assets by review status to quickly find any assets that must be looked at.

You can reset the review status of an asset at any time. To change the review status, click Mark as reviewed or Mark as not reviewed from the asset's overflow menu. To change the review status of several assets at once, select the assets, click More, and select Mark as reviewed or Mark as not reviewed. The review status of an asset is independent of what review status its columns have. You can also use APIs instead of the user interface to set the review status of assets. The links to these APIs are listed in the Learn more section.

When you do a bulk change of the review status, you might see a success message before the changes are actually complete depending on the volume of the requested changes. You might need to refresh the view several times before you see all changes applied.

Enrichment status

The enrichment status column can have these values:

Not analyzed

This asset was added after the last enrichment run.

Finished

Enrichment for this asset is complete. This status is also shown if the enrichment happened outside the current enrichment asset, for example, if the asset was profiled manually before it was added to this enrichment.

Failed

An error occurred during the enrichment.

Canceled

The job run for the enrichment was canceled.

You can sort or filter the result list by enrichment status. For sorting, the primary sort order is by status. Ascending order is canceled, failed, and finished. Depending on the general sort order, assets with the status Not analyzed are displayed unordered at the top or the end of the list.

Publish status

This column shows whether an asset was published to a catalog. Publish details such as the target catalog or the name of the publish job are available in the asset information in the side panel.

However, only the details of the most recent publish request are shown.

Column details

Access column and enrichment details by clicking the column name or by clicking View column details from the overflow menu. On the Details tab in the side panel, you find this information:

• Display name. If the enrichment options include the Expand metadata option, this section initially contains an alternative name for the column that was found through fuzzy matching. Fuzzy matching expands the source name based on a predefined glossary to provide a name that is easy to understand. The expanded name might already be assigned because the confidence was high enough or it is a suggestion that you can accept. At any time, you can edit the display name.

• Description. This section can contain a description for the column. If the enrichment options include the Expand metadata option, this section initially contains an AI-generated description. This description might already be assigned because the confidence was high enough or it is a suggestion that you can accept. At any time, you can edit the description.

• The context of the asset to which the column belongs in the Source section.

• Statistics about the data for each column such as the number of distinct values, the percentage of unique values, minimum, maximum, or mean, and sometimes the standard deviation in that column. The number of distinct values indicates how many different values exist in the sampled data for the column. The percentage of unique values indicates the percentage of distinct values that appear only once in the column.

  Depending on a column’s data format, the statistics vary slightly. For example, statistics for a column of data type integer have minimum, maximum, and mean values and a standard deviation value while statistics for a column of data type string have minimum length, maximum length, and mean length values.

• The frequency distribution of the values found and the number of missing values.

• The data format of the columns in the sampled rows.

• The asset owner.

More detailed results for each column are available in the column profile. To view that profile:

• Select View data profile from the column's overflow menu.
• Click the Open column profile icon next to Statistics or Formats in the column details.
• Click the View all link in the Statistics or Formats section. Whether this link is available depending on the number of results.

If the column is subject to a data protection rule, only a subset of this information is available: the description and the context

For Watson Query and watsonx.data view assets, all users are denied access to the profiling results to prevent accidental exposure of value distributions.

Governance information

Governance information for a column includes assigned and suggested business terms, which are listed in the Business terms column of the results, assigned and suggested data classes, which are shown in the Data class column, and assigned classifications, which are shown in the Classifications column. An automatically assigned data class is identified by a purple dot next to the data class name. For assigned terms, the dot indicates that at least one of the terms was automatically assigned.

Access detailed governance information for a column by clicking the column name, by clicking the View more link in the Business terms, Data class, or Classifications column, or by clicking View column details from the overflow menu. On the Governance tab in the side panel, you can manage term, data class, and classification assignments.

The same information is provided when you click the View more link that appears below the business term, data class, or classification when you hover over a specific column.

Data quality score

A data quality score is displayed only if at least one data quality check was applied to the column. Otherwise, a dash (—) is shown. Data quality scores are computed for each individual column in a data asset based on the results of the applied data quality checks. A setting in the data quality analysis results determines whether a column's quality core is considered for calculating the overall asset and dimension scores.

A delta value shows how the data quality score changed compared to the score from 90 days before the latest analysis:

• A green arrow pointing to the upper right () indicates that the data quality score went up.
• A red arrow pointing to the lower right () indicates that the data quality score went down.

To find columns with quality issues quickly, especially when the enrichment scope is large, you can filter the list by quality scores.

For details about data quality issues, select a column and click View data quality details from the overflow menu, or click the column's quality score.

For more information, see Data quality analysis results and Data quality scores.

Review status

Initially, the review status of all columns in the metadata enrichment is Not reviewed. After you review the enrichment results for a column, you can set its review status to Reviewed. Thus, everybody on the team is aware of what already was looked at and what still needs to be reviewed. If a later enrichment run updates the results of a column with the status Reviewed, the column's review status is set to Reanalyzed after review (). The review status does not change for updates that were found during metadata import.

Note that, for columns that are marked as reviewed, term assignments are not updated on reruns of an enrichment. For more information, see How new analysis results update existing term assignments.

You can reset the review status of a column at any time. To change the review status, click Mark as reviewed or Mark as not reviewed from the column's overflow menu. To change the review status of several columns at once, select the columns, click More, and select Mark as reviewed or Mark as not reviewed. A column's review status is independent of the review status that its containing asset has. You can also use APIs instead of the user interface to set the review status of columns. The links to these APIs are listed in the Learn more section.

When you do a bulk change of the review status, you might see a success message before the changes are actually complete depending on the volume of the requested changes. You might need to refresh the view several times before you see all changes applied.

If the built-in machine learning model for ML-based term assignment is used and is trained from project assets, columns that are marked as reviewed and have automatically assigned business terms serve as training data.

Filter the list of columns by review status to quickly find any columns that must be looked at.