Log Queries and Examples
The platform generates platform logs from the proxy. You need the proxy URL to set up your integrated development environment (IDE) when running a query on the proxy. You can get this from within the platform:
Click View Exchange to open the data exchange.
Open the Data Plane tab.
Open the Data Proxy tab to see the proxy host details.
Next, run a query from an IDE, such as VisualStudio or IntelliJ IDEA to produce a log. The proxy produces a two-audit event record; one of action type user.authentication and another of action type policy.resolution.
Note that the query runs against database metadata, not the platform metadata.
For the query audit event, you can set it to be verbose or non-verbose with an application property of: privitar.jdbcproxy.queryengine.verboseAuditLog. By default it is false. If you set it to true, the query audit events will also include information about the request query and the transformations applied to the field.
The audit event does not include information about the execution nor status of the request (query).
Audit Log Field Descriptions
Field | Description |
|---|---|
| A unique audit ID, generated for each event. |
| A system event which generates this audit record. The format of this field is |
| Event type generating this event. For example, |
| The time that the event generated the audit record. |
| The username, group name, or system name of the entity that triggered the event. For certain events, this can be NULL. |
| Whether initiator is a user or system_user. |
| The tenant ID where the event takes place. For certain events, this can be NULL. |
| The system observing the event. |
| Outcome of event is either |
| Optional error message when the outcome is |
| The target system ID observing the event. When running on Kubernetes this is the node ID. In other environments this is the ID of the data plane. |
| The target system type. |
| The impact of an event on the system.
|
| Optional. When running in Kubernetes, this is the namespace and pod where the event takes place. |
| The ID to link to the incoming request ID that generates this event. For workflow events, the |
| Optional. This is the response data that the event generated. |
| Optional. This is the request data for the event. |
Policy Resolution Attachment Detail
The following explains common attributes that you're likely to see in the attachments section of the JSON view of a policy resolution log file:
correlationId—A common identifier for the request or related group of requests.requestData—Provides context and result detail for the policy resolution event.context—The metadata used for policy resolution.connection—A configuration for connecting to and reading data from a data source, such as a JDBC connection string.id—The unique ID of the data source connection used to access the requested data.name—The name of the data source connection used to access the requested data.
groupIds—The identifier of the group who requested the data.project—The consumption project used to generate the request.id—The unique ID of the consumption project used to generate the request.name—The name of the consumption project used to generate the request.
request—Variable based on the type of request. Eithermetadataorquery.metadata—Issued when data owner requests metadata for a data source (alwaysget columns).query—Issued when the data consumer queries data (always aselectstatement).
userId—The identifier of the user who requested the data.userName—The username of the user who requested the data.
fieldTriggers—Describe how policies were triggered.fields—Fields that were included in the query.assetId—Unique identifier for the asset.assetName—Name of the asset as described in the data exchange.catalog—The dataset name.field—The field that was included in the query.schema—The name of the schema that the table and field belong to.table—The name of the table that the field belongs to.
trigger—Describes which condition was triggered.policyId—Unique identifier for the policy that was triggered.policyName—The name of the policy that was triggered.ruleCondition—The rule condition that caused the policy to trigger.ruleId—The unique identifier of the rule that caused the policy to trigger.ruleName—The name of the rule that caused the policy to trigger.
filters—The record-level access control (RLAC) filters that were triggered.expression—The RLAC rule that was triggered.field—The field included in the RLAC rule.assetId—Unique identifier for the asset.assetName—Name of the asset as described in the data exchange.catalog—The dataset name.field—The field that was included in the query.schema—The name of the schema that the table and field belong to.table—The name of the table that the field belongs to.
request—Variable based on the type of request. Eithermetadataorquery.metadata—Issued when data owner requests metadata for a data source (alwaysget columns).query—Issued when the data consumer queries data (always aselectstatement).
transformations—The masking transformations that were applied based on the triggered policies.field—The field that was masked by the transformation logic.assetId—Unique identifier for the asset.assetName—Name of the asset as described in the data exchange.catalog—The dataset name.field—The field that was included in the query.schema—The name of the schema that the table and field belong to.table—The name of the table that the field belongs to.
singleFieldTransformation—The type of transformation applied to the field.transformationName—The name given when the transformation was created.constant,drop,generalizeDate,numericRegex,redactWithNull,regex,retain, ortruncate—The transformation type applied to the field.consistencyGroupId—The unique identifier of the group of tokens used in consistent tokenization.consistent—truemeans that consistent tokenization was applied.falsemeans means that consistent tokenization was not applied.regex—The regular expression that the platform applied a to the field.watermarkingRegex—The watermarking regular expression that the the platform applied to the field.value—The value applied to the field for the Constant Text Value (constant) transformation type.length—The number of characters to preserve or delete when applying the Truncate transformation type.preserveSelectedCharacters—truemeans the platform preserved the characters indicated bylengthwhen applying the Truncate transformation type.falsemeans the platform deleted the characters indicated bylengthwhen applying the Truncate transformation type.truncateFromStart—truemeans the platform applied the Truncate transformation type from the beginning of the string.falsemeans the platform applied the Truncate transformation type from the end of the string.
cellLevelTransformation—The type of transformation applied to the field.else—The condition applied if none of theifAndElseIfsconditions were true.singleFieldTransformation—The type of transformation applied to the field.transformationName—The name given when the transformation was created.constant,drop,generalizeDate,numericRegex,redactWithNull,regex,retain, ortruncate—The transformation type applied to the field.consistencyGroupId—The unique identifier of the group of tokens used in consistent tokenization.consistent—truemeans that consistent tokenization was applied.falsemeans means that consistent tokenization was not applied.regex—The regular expression that the platform applied a to the field.watermarkingRegex—The watermarking regular expression that the the platform applied to the field.value—The value applied to the field for the Constant Text Value (constant) transformation type.length—The number of characters to preserve or delete when applying the Truncate transformation type.preserveSelectedCharacters—truemeans the platform preserved the characters indicated bylengthwhen applying the Truncate transformation type.falsemeans the platform deleted the characters indicated bylengthwhen applying the Truncate transformation type.truncateFromStart—truemeans the platform applied the Truncate transformation type from the beginning of the string.falsemeans the platform applied the Truncate transformation type from the end of the string.
ifAndElseIfs—When true, a condition for which the platform applies a transformation.logicalExpression—Contains the expression and the affected fields.expression—The logic for evaluating whether the platform applied a cell-level transformation.field—A field that was masked by a cell-level transformation.assetId—Unique identifier for the asset.assetName—Name of the asset as described in the data exchange.catalog—The dataset name.field—The field that was included in the query.schema—The name of the schema that the table and field belong to.table—The name of the table that the field belongs to.
transformation—The masking transformation that was applied based on the triggered policies.singleFieldTransformation—The type of transformation applied to the field.transformationName—The name given when the transformation was created.constant,drop,generalizeDate,numericRegex,redactWithNull,regex,retain, ortruncate—The transformation type applied to the field.consistencyGroupId—The unique identifier of the group of tokens used in consistent tokenization.consistent—truemeans that consistent tokenization was applied.falsemeans means that consistent tokenization was not applied.regex—The regular expression that the platform applied a to the field.watermarkingRegex—The watermarking regular expression that the the platform applied to the field.value—The value applied to the field for the Constant Text Value (constant) transformation type.length—The number of characters to preserve or delete when applying the Truncate transformation type.preserveSelectedCharacters—truemeans the platform preserved the characters indicated bylengthwhen applying the Truncate transformation type.falsemeans the platform deleted the characters indicated bylengthwhen applying the Truncate transformation type.truncateFromStart—truemeans the platform applied the Truncate transformation type from the beginning of the string.falsemeans the platform applied the Truncate transformation type from the end of the string.
dataPlane—Provides context about the data plane on which the policy resolution event occurred.id—The unique identifier of the data plane.
rlacFilterBehaviour—The default behavior of the RLAC filter as set by the exchange administrator; either "DENY_ACCESS" or "ALLOW_ACCESS".
responseData—Not currently used.source—The data plane through which the audit event was issued.namespace—The Kubernetes namespace of the data plane through which the audit event was issued.pod—The Kubernetes pod on which the audit event was issued.
Example of the User Info Audit
{
"header": {
"id": "00f5fc6b-a1d2-4db1-be44-9d7360a25176",
"event": {
"eventTime": "2022-11-04T12:36:49.439Z",
"eventType": "ACTIVITY",
"action": "user.authentication"
},
"initiator": {
"id": "29bd6d2a-10a3-44aa-8de7-1f264e22e21c",
"typeURI": "User",
"tenant": "fff7a844-f732-4a65-8280-912eae8eb8d0"
},
"observer": {
"id": "target"
},
"status": {
"outcome": "SUCCESS"
},
"target": {
"id": "01234567-89ab-cdef-0123-456789abcdef",
"typeURI": "service/compute"
},
"severity": "WARNING"
},
"attachments": [
{
"source": {
"namespace": "dpp-proxy",
"pod": "dynamic-proxy-b48ad3d7-7b7c-431f-ad5d-633e5641c0cd-59c677blq52m"
},
"correlationId": "93429272-b9e7-4452-b799-dfeddf5e3a47",
"requestData": {
"context": {
"project": {
"id": "5a078d49-1bc3-4a6f-9caa-1d622f18e7cc"
},
"requestAttributes": {
"request.project": "5a078d49-1bc3-4a6f-9caa-1d622f18e7cc"
}
}
}
}
]
}You can see that "action" tells us what type of audit log it is.
Example of the Non-verbose Query Audit Event
{
"header": {
"id": "40012d8a-f002-4fcf-87cd-549e66d57634",
"event": {
"eventTime": "2022-11-07T06:46:13.024082Z",
"eventType": "ACTIVITY",
"action": "policy.resolution"
},
"initiator": {
"id": "ba992d9f-2c0c-476e-9584-9e8ef923a5bf",
"typeURI": "User",
"tenant": "bc5bbfa3-e07c-4688-8644-a516afe40c1b"
},
"observer": {
"id": "target"
},
"status": {
"outcome": "SUCCESS",
"reason": null
},
"target": {
"id": "ip-10-18-122-83.eu-west-1.compute.internal",
"typeURI": "service/compute"
},
"severity": "WARNING"
},
"attachments": [
{
"source": {
"namespace": "dpp-8193-common-audit-dp",
"pod": "dynamic-proxy-b48ad3d7-7b7c-431f-ad5d-633e5641c0cd-59c677blq52m"
},
"correlationId": "b93069e5-676b-41ce-9c2d-c69c2b5d9360",
"responseData": null,
"requestData": {
"transformations": [
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
}
]
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "lastname"
}
]
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "id"
}
]
}
],
"filters": [
{
"field": {
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
},
"expression": "firstname <> 'AAAAA'"
}
]
}
}
]
}Example of the Verbose Query Audit Event for a SQL Query
Turn auditing on, is true by default: privitar.audit.enabled=true
Configure auditing to emit records to Fluent Bit: privitar.audit.auditServiceUrl=true
{
"header": {
"id": "00f5fc6b-a1d2-4db1-be44-9d7360a25176",
"event": {
"eventTime": "2022-11-04T12:36:49.439Z",
"eventType": "ACTIVITY",
"action": "policy.resolution"
},
"initiator": {
"id": "successfulUser",
"typeURI": "User",
"tenant": "fff7a844-f732-4a65-8280-912eae8eb8d0"
},
"observer": {
"id": "target"
},
"status": {
"outcome": "SUCCESS",
"reason": null
},
"target": {
"id": "01234567-89ab-cdef-0123-456789abcdef",
"typeURI": "data_plane"
},
"severity": "WARNING"
},
"attachments": [
"source": {
"namespace": "dpp-proxy",
"pod": "dynamic-proxy-b48ad3d7-7b7c-431f-ad5d-633e5641c0cd-59c677blq52m"
},
{
"userId": "proxy-user",
"request": {
"query": "select * from library.authors"
},
"correlationId": "93429272-b9e7-4452-b799-dfeddf5e3a47",
"responseData": null,
"requestData": {
"transformations": [
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "lastname"
}
],
"singleFieldTransformation": {
"retain": {}
}
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
}
],
"singleFieldTransformation": {
"regex": {
"regex": "[A-Za-z]{1,15}"
}
}
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "id"
}
],
"singleFieldTransformation": {
"retain": {}
}
}
],
"filters": [
{
"field": {
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
},
"expression": "firstname <> 'AAAAA'"
}
]
}
}
]
}Example of the Verbose Query Audit Event for a Metadata Request
{
"header": {
"id": "00f5fc6b-a1d2-4db1-be44-9d7360a25176",
"event": {
"eventTime": "2022-11-04T12:36:49.439Z",
"eventType": "ACTIVITY",
"action": "policy.resolution"
},
"initiator": {
"id": "proxy-user",
"typeURI": "User",
"tenant": "fff7a844-f732-4a65-8280-912eae8eb8d0"
},
"observer": {
"id": "target"
},
"status": {
"outcome": "SUCCESS"
},
"target": {
"id": "01234567-89ab-cdef-0123-456789abcdef",
"typeURI": "data_plane"
},
"severity": "WARNING"
},
"attachments": [
{
"userId": "proxy-user",
"request": {
"metadata": "get tables"
},
"correlationId": "93429272-b9e7-4452-b799-dfeddf5e3a47",
"requestData": {
"transformations": [
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
}
],
"singleFieldTransformation": {
"regex": {
"regex": "[A-Za-z]{1,15}"
}
}
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "id"
}
],
"singleFieldTransformation": {
"retain": {}
}
},
{
"field": [
{
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "lastname"
}
],
"singleFieldTransformation": {
"retain": {}
}
}
],
"filters": [
{
"field": {
"catalog": "acmedb",
"schema": "library",
"table": "authors",
"field": "firstname"
},
"expression": "firstname <> 'AAAAA'"
}
]
}
}
]
}Logs for Possible Failures
There are also audit logs for some possible failures, such as when user authentication has failed:
{
"header": {
"id": "00f5fc6b-a1d2-4db1-be44-9d7360a25176",
"event": {
"eventTime": "2022-11-04T12:36:49.439Z",
"eventType": "ACTIVITY",
"action": "user.authentication"
},
"initiator": {
"id": "user@gmail.com",
"typeURI": "User",
"tenant": "fff7a844-f732-4a65-8280-912eae8eb8d0",
"sourceIp": null
},
"observer": {
"id": "target"
},
"status": {
"outcome": "FAILURE",
"reason": {
"message": "Received error while trying to authenticate username user@gmail.com: Server Error"
}
},
"target": {
"id": "01234567-89ab-cdef-0123-456789abcdef",
"typeURI": "data_plane"
},
"severity": "WARNING"
},
"attachments": [
]
}This log shows a policy retrieval failure.
{
"header": {
"id": "00f5fc6b-a1d2-4db1-be44-9d7360a25176",
"event": {
"eventTime": "2022-11-04T12:36:49.439Z",
"eventType": "ACTIVITY",
"action": "policy.resolution"
},
"initiator": {
"id": "proxy-user",
"typeURI": "User",
"tenant": "fff7a844-f732-4a65-8280-912eae8eb8d0",
"sourceIp": null
},
"observer": {
"id": "target"
},
"status": {
"outcome": "FAILURE",
"reason": {
"message": "Failed to retrieve policies. Internal server error"
}
},
"target": {
"id": "01234567-89ab-cdef-0123-456789abcdef",
"typeURI": "data_plane"
},
"severity": "WARNING"
},
"attachments": []
}