Skip to main content

Process: Event Processing Workflow

What is this process and why do I care about it?​

Event Processing Workflow delivers the core value of Pyrae Policy Engine. It is responsible for executing the Observers, evaluating Policies, storing changes for Violations, and triggering Integrations.

Prerequisites​

  1. AWS Account Onboarding
  2. At least some Teams
  3. At least some Resources in your AWS account
  4. At least some Policy Signatures

Lifecycle​

Workflow/Process Diagram​

A diagram showing a preview of the steps outlined in the subsequent lifecycle steps

Execution Trigger​

The workflow is invoked when:

  1. a Resource has changed, and AWS CloudTrail has notified us of that change.
  2. one of your developers with access to the Pyrae UI clicks the "Observe Now" Button
  3. a Temporal Policy is ready to run

Receive an instruction to observe a ResourceURN​

The Workflow is triggered with an object that contains details on what needs to be observed.

In the case of CloudTrail or a manual execution, it includes the actual ResourceUrns.

In the case of a temporal policy, we don't have the actual Resources, just the Policy. This step identifies the corresponding Resources.

Perform the Owner Identification Process​

Later steps in the workflow rely on the resource having an up-to-date owner. For example, Policies only run for a specific Resource when that Policy has been signed by the Team that owns that Resource.

So, this is a critical early step which identifies and stores the owner of each Resource.

This is accomplished by looking at the Account object. The Account object has a field named ownerMatchingStrategy, which specifies how the owner should be identified.

The valid strategies are:

  • ALL_INTO_ONE_BUNDLE - Consolidates resources into one bundle, specified in defaultOwnerBundleUrn
  • GET_BUNDLE_FROM_TAG - Identifies the owner bundle name or bundle URN from the pyrae:OwnerTag on the AWS resource.
  • EXECUTE_CUSTOM_LAMBDA - Runs a custom Lambda that satisfies the Owner Identification Lambda Contract.

Delay if the Resource is not ready​

Some Resources may be in transient states when the workflow begins, and may not be ready for a scan. For example, an RDS instance may be undergoing an upgrade. This would interfere with the Policy Engine by creating an inconsistent Observation of the resource, which may cause Observances to open or close unexpectedly.

Pyrae Policy Engine reduces this problem by checking every Resource's readiness immediately, and rechecking every 15 minutes for up to 1 day. If the Resource hits the 1 day retry limit, or an error occurs in readiness checking, the Resource continues to the next step anyway. This means that the delay doesn't cause any scans to be dropped.

Calculate which Observers need to run​

Each Resource has an URN. We take that URN, deconstruct it into its pieces (Account Type, Account ID, Service Name, ...) and search for ObserverMatchingRules that match it. Each ObserverMatchingRule refers to a single Observer, which is the list we've been looking for.

We search for ObserverMatchingRules deployed into the Org that owns the Resource, and we search in all of the Linked Orgs too. This is because the Org Link means that the Observer crosses the Org boundary.

Observers run without regard for Policy Signatures; Observers run even if the team hasn't signed a related Policy. This is so that you have a consistent view of the history of a Resource, which is useful when making a judgement as to whether a Policy should be signed. It's very useful to know that N resources are already in violation of a policy that you have not yet adopted, which is not possible without always executing Observers.

Run those Observers​

Observers are AWS Lambda Functions that satisfy the Observer Contract. In order to prepare the input for the Observer Contract, the Event Processing Workflow must obtain credentials for each AWS Account that the Observer will need to access.

These credentials are an IAM AssumeRole Session to a Role named PyraeAccessRole, which was provisioned in AWS Account Onboarding. The Session is constrained by a Session Policy to reduce observer blast radius; Observers only have acess to the API operations necessary, even if the Role grants much more. The Session Policy is specified in the Observer's IamSession field.

Store the Observances​

As part of the Observer Contract, the Observer outputs a set of Observations, typically one Observation per Resource. Those need to be persisted for the UI and for use in the later steps.

Identify Signed Policies for each Observation​

As we established in an earlier step, each Resource has a clearly identified Owning Bundle. We can climb up the Bundle tree to identify a singular owning Team, exactly one Team for each Resource.

Each Team has an optional ParentTeamUrn, forming a Team tree. We'll climb up the tree to collect a list of Teams.

Each Team can independently sign Policies. So, We need to execute every Policy that is Signed by any of the parent Teams of the Resource.

Run those Policies​

Each Policy satisfies the Policy Contract. At its core, the Policy Contract takes in a list of Observances (all observing the same Resource), and outputs a list of Violations. The Polcy Contract accomplishes this by evaluating an expression.

Compute and Store Violation Changes​

The Policy Contract does not consider the current state of Violations. Policies output the desired state of violations. This step's job is to reconcile the desired state with the actual state by opening new violations and closing existing violations.

Identify Integrations that need to run​

We inherit the list of violation changes from the prior step -- where a new violation was opened, or an existing violation was closed, but excluding where the actual state matched the desired state.

For each change, we need to identify all integrations that need to be executed. To do that, we search for IntegrationViolationTriggerRules that match the metadata of the change. IntegrationViolationTriggerRules can match on the ResourceUrn, Direct Owner TeamUrn, any parent TeamUrn, or the Policy Urn.

It's possible for a change to match multiple integrations.

Run those integrations​

Each integration internally contains a mapping between the changeset and a series of API/RPC operations on the underlying integration. For the time being, this mapping is an internal detail, but it may be an exposed contract in the future.

Pyrae Policy Engine

If you're concerned about monitoring policies in your organization, our Policy Engine can monitor your infrastructure and provide actionable alerts.

Contact Us