How T-Systems Enabled AWS Config at Scale for Deutsche Telekom IT’s Landing Zone

By Abdullatif Rashdan, Associate Consultant – AWS Professional Services
By Steffen Mazanek, Lead Solution Architect – T-Systems
By Manuel Kellner, DevSecOps Engineer – Deutsche Telekom IT

T-Systems

In this post, we will explain how T-Systems has enabled AWS Config at scale on Deutsche Telekom IT’s Amazon Web Services (AWS) landing zone, which is a brownfield environment with more than 600 AWS accounts.

AWS Config is a governance and compliance service that enables you to assess, audit, and evaluate the configurations of your resources. The main motivator for Deutsche Telekom IT (DTIT) using AWS Config is that developing custom compliance checks using custom tools was time consuming, unscalable, and came with a large operational overhead.

Deutsche Telekom IT preferred AWS Config as it is:

• Natively integrated with the other AWS services.
• Provides a large set of AWS managed rules based on best practices.
• Eliminated the operational overhead of managing a custom tool for compliance checks.
• Shortens the time of the development of custom compliance checks for new AWS services.
• Gives an aggregated view of all member accounts in the AWS Organization through the use of AWS Config Aggregators.

When T-Systems tried to enable AWS Config for the entire environment, it realized the service was not centrally governed. Some AWS member accounts enabled AWS Config with their own configuration for local use cases. For example, local vs. centralized Amazon Simple Storage Service (Amazon S3) bucket to store configuration items.

As T-Systems and Deutsche Telekom IT wanted to gain insights and visibility on the usage of AWS Config on member accounts, they engaged with AWS Professional Services to help accelerate the process by automating the discovery, cleanup, and enablement of AWS Config at scale.

After enabling AWS Config across all accounts, T-Systems obtained a comprehensive view on resource compliance in the environment by setting up an Aggregator in the management account. The team realized the value of AWS Config, and believes in the future it will be faster and scalable to evolve AWS security standards and maintain resource compliance using this service as it is natively integrated with AWS.

T-Systems is an AWS Premier Tier Services Partner and Managed Cloud Services Provider (MSP) that helps clients such as Deutsche Telekom set up their landing zones and networks, and by supporting hubs and applications in adopting AWS. With a footprint in more than 20 countries, T-Systems is one of the world’s leading vendor-independent providers of digital services headquartered in Europe.

Deutsche Telekom IT is the internal IT service provider of Deutsche Telekom AG and is responsible for the design, development, and operation of owned and transferred IT systems supporting Deutsche Telekom AG business processes.

Initial Setup

When DTIT started its cloud journey with AWS in 2019, T-Systems built up a landing zone for DTIT that includes components such as account factory, central logging, and service catalog products. Since then, DTIT’s DevSecOps team is implementing specific security requirements of telecom industry into the environment.

DTIT’s DevSecOps team used an open source tool to develop security checks on AWS. The initial setup sends all AWS CloudTrail events from the member accounts to the DevSecOps account.

AWS Lambda functions in the DevSecOps account are triggered by CloudTrail events. They parse the event and evaluate it against a set of compliance and security rules, notifies users about violating resources, and in some cases automatically remediates risks.

Figure 1 – Initial event-driven setup.

Motivation for AWS Config and Target Architecture

After about two years in production, DTIT started to face limitations in its setup. The team figured that developing custom security and compliance checks was time-consuming, unscalable, and involved lots of manual work.

To develop a single security check for an AWS service, DTIT would identify a security threat and analyze all related service events. That slowed the team down, as they wanted to focus on making the environment secure and not parsing events. All of that made DTIT look for a solution that is managed, natively integrated with AWS services, and works well within AWS Organizations.

Initially, the setup served its purpose to perform security and compliance checks, but the security team wanted to have a central view of the compliance status of all AWS accounts in the environment. Furthermore, DTIT realized the current setup lacked a dashboard where workload owners could see the compliance status of their environment.

DTIT found that AWS Config fulfills all of these requirements. It’s a managed service that is constantly monitoring the resources of an account for changes and can evaluate them against a given set of rules. By using AWS Config, DTIT eliminated the time-consuming process of analyzing all service events, followed by developing custom checks. AWS Config does that heavy lifting on behalf of you.

AWS Config offers a huge library of AWS managed rules known as conformance packs, and you can customize the conformance packs so it includes a combination of custom and AWS managed rules. This accelerated the development of DTIT security controls, as the team did not need to start everything from scratch.

AWS Config simplifies the deployment of compliance and security rules across all member accounts in your organizations. This allowed DTIT to centrally deploy, update, and delete rules across all member accounts in the environment. To learn more, see the AWS documentation.

Once AWS Config is enabled in an AWS account, workload owners can see the compliance status of their account which was not possible in the previous setup.

The DTIT security team’s requirement of obtaining a complete view on all member accounts was satisfied by enabling multi-account, multi-region data aggregation, which provided an aggregated view on all member accounts.

As a result, DTIT decided to adopt AWS Config and gradually transfer custom controls to AWS Config rules.

Challenges

Before enabling AWS Config, DTIT had a brownfield environment with over 600 AWS accounts in total. Some of them had existing AWS Config recorders and delivery channels created by end users for local use cases in their AWS accounts.

T-Systems had to discover and examine which accounts had AWS Config enabled and what the use cases were. Due to the large volume of AWS accounts, manual operations were deemed infeasible and the team had to automate the entire process.

As there is a limit of one recorder per region for AWS Config, that meant previously-enabled recorders should be deleted before starting a centralized enablement. These recorders were not following a naming convention, nor were they created with proper configurations.

Issues with existing recorders are:

• Undesired configurations, for example, setting up a local Amazon S3 bucket for configuration items (CIs) or, generally, not being aligned with the organization security goals.
• Recording inessential set of AWS resources.
• AWS Config is enabled but not utilized, such as for AWS accounts that are used for testing and development purposes.

The challenge was how T-Systems could enable AWS Config across all member accounts and govern it centrally. In the next section, we explain how T-Systems prepared the environment by discovering and cleaning up AWS Config recorders, enabling AWS Config centrally using AWS StackSets, and governing it using Service Control Policies (SCPs) to prevent disabling or deleting recorder.

Enabling AWS Config at Scale in a Brownfield Environment

Before enabling AWS Config at scale, the first step is to discover existing recorders in the brownfield environment, followed by cleaning them up while skipping some exception accounts.

To automate this process of deleting AWS Config recorders, T-Systems used AWS Lambda functions:

1. ConfigDiscovery Lambda function: Discovers the existing AWS Config recorders in the environment and stores the results in an S3 bucket.
   • The results consist of snapshot of all current AWS Config rules and summary spreadsheet for all accounts.
2. ConfigCleanup Lambda function: Deletes AWS Config recorders and delivery channels.

Lambda functions are packaged using the AWS Serverless Application Model (SAM). This makes it easier for customers who want to enable and govern AWS Config centrally.

Figure 2 – SAM package used for discovery and cleanup of AWS Config recorders.

Prerequisites

• List of excluded account you wish to ignore in discovery and cleanup phase.
• SAM CLI installed and configured with credentials to the management account.

The following steps are performed to discover and clean AWS Config recorders:

• Deploy the SAM package in the management account.
• Provision cross-account AWS Identity and Access Management (IAM) role using AWS StackSets.
• Invoke the discovery function.
• Invoke the cleanup function.
• Enable AWS Config at scale using AWS StackSets.
• Apply Service Control Policies to protect AWS Config recorders from deletion.

Deploying the SAM Package

1. Download the SAM package for AWS Config discovery and cleanup.
2. Unzip the package.
3. Navigate to the root directory.
4. Run the following command in the management account:

sam deploy --stack-name STACK_NAME --region AWS_REGION --capabilities CAPABILITY_NAMED_IAM --resolve-s3

Figure 3 – Running the ‘sam deploy’ command.

5. Once completed, SAM CLI displays the stack output section. You will use those functions in the next section to perform discovery and cleanup.

Figure 4 – The ‘sam deploy’ command shows the CloudFormation stack outputs when completed.

Deploying Cross-Account IAM Role

AWS Lambda functions assume an IAM role across all member accounts to perform discovery and cleanup. It allows the following actions:

        Statement: 
          - Effect: "Allow"
            Action: config:Describe*
            Resource: "*"
          - Effect: "Allow"
            Action: sts:GetCallerIdentity
            Resource: "*"          
          - Effect: "Allow"
            Action: config:DeleteConfigurationRecorder
            Resource: "*"
          - Effect: "Allow"
            Action: config:DeleteDeliveryChannel 
            Resource: "*"

The IAM role trusts the discovery and cleanup functions in the SAM package deployed in the previous step. It allows the functions to get information about AWS Config configurations and delete recorders and delivery channels.

1. Download the IAM role.
2. Follow the steps on Create a stack set to deploy this IAM role to all member accounts where you want to run discovery and cleanup.
3. Enter the management account ID in the parameter section for the stack set.

Invoking the Discovery Lambda Function

This function iterates through all member accounts and discovers if there’s an AWS Config recorder configured in that account. If yes, then it checks if there’s any AWS Config rules and conformance packs deployed in that account and creates a snapshot for them in the result S3 bucket.

Note that the result bucket is provisioned as part of the SAM package deployed in the first step.

1. Navigate to AWS Lambda and open the discovery function.
2. Invoke the function with the following event:

The discovery function takes two parameters:

1. region (string): specify which region to run the discovery.
2. excluded_accounts (list of integer account IDs): This is the list of accounts which you don’t want to run a discovery on them. Note that the discovery can be performed on several regions without the need to deploy the SAM package to every region by changing the region parameter in the invocation event.
3. Once finished, open the result S3 bucket to view it.

Figure 5 – Output of the discovery function in the S3 bucket.

AWS accounts where AWS Config rules and conformance packs were configured will have a standalone folder that has a snapshot of them.

The aggregated summary for that region will be stored in AWSConfigDiscovery/AWS_REGION/ AWSConfigDiscoveryResult.csv:

Figure 6 – Summary output from discovery function shows the status of AWS Config recorders in each account.

It tells you which accounts has AWS Config enabled, and basic account information such as account ID, name, and email which can help facilitate communicating with the account owners to inform them about the change.

4. After examination and inspection:
   • Communicate changes to the users.
   • Determine which accounts should be skipped and save the list prior to cleanup step.

Invoking the Cleanup Lambda Function

This function iterates through all member accounts and checks if there’s an AWS Config recorders setup. If yes, then it deletes the AWS Config recorder and delivery channel while skipping a list of accounts.

Follow these steps to invoke the cleanup Lambda function to delete the AWS Config recorders and delivery cannel in your landing zone:

1. Navigate to AWS Lambda and open the cleanup function.
2. Invoke the function with the following event:

• • Be sure to specify which accounts you wish will be excluded from the cleanup.

The cleanup function takes two parameters:

3. region (string): Specify which region to delete the AWS Config recorders and delivery channel.
4. excluded_accounts (list of integer account IDs): This is the list of accounts which you don’t wish to delete the AWS Config recorders and delivery channel.

The cleanup can be performed on several regions without the need to deploy the SAM package to every region by changing the region parameter in the invocation event.

The response of the cleanup function will tell you on how many accounts the automation succeeded, failed, excluded, and the total number of accounts.

{
  "TotalAccountCount": 8,
  "SuccessAccountsCount": 7,
  "ExcludedAccountsCount": 1,
  "FailedAccountsCount": 0,
  "FailedAccountList": []
}

After running this function successfully, all AWS Config recorders should be deleted. Now, you’re ready for the enablement of AWS Config.

Note that you can view the logs of the Lambda function to see the results on account level.

Enabling AWS Config at Scale Using AWS StackSets

You can leverage this AWS CloudFormation StackSets template – Enable AWS Config – to enable AWS Config in your organization with central configurations.

Applying SCPs to Prevent Altering AWS Config Configuration

Service Control Policies are a type of organization policy you can use to manage permissions in your organization.

In this scenario, T-Systems wanted to enable AWS Config for across all member accounts in the brownfield environments with central configurations. However, what the team didn’t want was for the account owners to disable, delete AWS Config recorder and its delivery channel.

That’s why they leveraged the following SCP:

{
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Deny”,
      “Action”: [
        “config:DeleteConfigurationRecorder”,
        “config:DeleteDeliveryChannel”,
        “config:StopConfigurationRecorder”
      ],
      “Resource”: “*”
    }
  ]
}

The above SCP allows account users to create and deploy their own AWS Config rules and conformance packs. This is done intentionally as DTIT wanted account owners to be able to deploy custom rules for their use cases while they are deploying organization-wide conformance packs to enforce compliance for our security controls on the entire landing zone.

Learn more about this in Managing Conformance Packs Across All Accounts in Your Organization.

Summary

This post showcased how moving to AWS Config enabled Deutsche Telekom IT (DTIT) to accelerate and improve its security standards in the organization. In a large brownfield environment, AWS Config gave DTIT a comprehensive and aggregated compliance view across all members accounts in the organization.

Previously and locally enabled AWS Config recorders may be created with undesired configuration that does not align with your organization security goals. In this case, T-Systems wanted all AWS Config recorders to follow the same standard.

AWS Professional Services helped T-Systems and DTIT automate the process of discovering, cleaning up AWS Config recorders, and then creating recorders with unified configurations. Furthermore, Deutsche Telekom IT has a new foundation where it uses conformance packs that are further modified to meet security standards.

.

.

T-Systems – AWS Partner Spotlight

T-Systems is an AWS Premier Tier Services Partner and MSP that helps clients set up their landing zones and networks, and by supporting hubs and applications in adopting AWS.

Contact T-Systems | Partner Overview