OneLinersCommand workbench
Guides
Networking & DNS / Security / DevOps & CI/CD

Provision a hardened AWS EC2 web server in a private VPC

Deploy a private Amazon Linux 2023 web host with CloudFormation change sets, no public IP or SSH, SSM endpoints and Session Manager, explicit network paths, least-privilege role, IMDSv2, encrypted EBS, audit, drift, and rollback.

240 min14 stepsChanges system stateRevision 2
Save or explore
Save to collectionCreate a collection in the sidebar first.
0 of 14 steps completed
Goal

Create an evidence-backed EC2 host that can be administered and monitored without public management exposure or broad cloud/network authority.

Supported environments
  • Amazon EC2 current regional API
  • Amazon Linux 2023 reviewed AMI
  • AWS CLI v2
  • CloudFormation 2010-09-09 template format
Prerequisites
  • Governed AWS account and region Federated short-lived operator role, CloudTrail/Config/SCP guardrails, approved region/AZs, quotas, account alias, and cost/change owners.aws sts get-caller-identity --query Arn --output text
  • Approved private network allocation Non-overlapping VPC/subnet CIDRs, internal DNS/resolver plan, client security group, regional S3 prefix list, and no requirement for public/NAT routing.
  • Approved image and private management Reviewed AL2023/golden AMI, SSM Agent, endpoint service availability, Session Manager logging/preferences, and emergency serial-console procedure.
  • Protected infrastructure repository Reviewed CloudFormation template/parameters, policy validation, secret scanning, exact change set approval, rollback owner, and no durable host data.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A CloudFormation-managed AWS VPC containing private application subnets without automatic public addresses or a default internet route, plus private AWS service endpoints required for Systems Manager administration.
  • A hardened Amazon Linux 2023 EC2 web host with no inbound SSH, Session Manager access, IMDSv2 required, encrypted gp3 root storage, termination protection, a non-default application port, explicit security-group ingress and egress, and a narrowly scoped instance role.
  • A preview-first deployment workflow using AWS CLI identity checks, template validation, CloudFormation change sets, resource drift and rollback evidence, Session Manager verification, CloudTrail, and service-level health.
Observable outcome
  • The instance has no public IPv4 address, no internet gateway/default route, no port 22 rule, no unrestricted ingress, and no unrestricted outbound internet path.
  • Systems Manager reports the instance Online through VPC endpoints; operators start attributable sessions without distributing SSH keys or exposing an SSH daemon. CloudTrail records session lifecycle, while compatible interactive shell sessions can additionally stream content to the configured log group.
  • Metadata requests without a token fail, token-authenticated metadata works only at hop limit one, every EBS volume is encrypted, and the web service answers only through the approved client security group and port.
  • CloudFormation change sets expose every create/update/delete/replace before execution, stack rollback is enabled, CloudTrail records control-plane activity, and teardown requires explicit protection removal and evidence retention.

Architecture

How the parts fit together

CloudFormation declares a dedicated VPC and two private subnets across availability zones. Their route tables contain only local and S3 gateway routes; there is no internet gateway, NAT gateway, or automatic public IPv4 assignment. Interface VPC endpoints for Systems Manager, Session Manager messages, and CloudWatch Logs place private service interfaces in the VPC with private DNS; an EC2 Messages endpoint is added only when the selected region and SSM Agent generation demonstrably require it. The instance security group accepts the web port only from an approved client or load-balancer security group. Egress permits TLS to endpoint interfaces and TLS to the regional S3 managed prefix list for signed Amazon Linux package content; it does not permit 0.0.0.0/0. AmazonProvidedDNS traffic cannot be filtered by security groups, so domain filtering is a separate Route 53 Resolver DNS Firewall control rather than a misleading port 53 rule. The EC2 instance runs Amazon Linux 2023 from an approved SSM public parameter, requires IMDSv2, encrypts its gp3 root volume, uses no key pair, and has API termination protection. A custom instance role enumerates only current SSM channel and logging actions; unavoidable Resource-star operations are documented because those APIs do not support resource-level ARNs, never used as an action/principal wildcard shortcut.

Private VPC and subnetsProvide isolated addressing and routing. No subnet assigns public addresses and no route sends arbitrary traffic to the internet.
Interface and gateway endpointsCarry Systems Manager control channels, logs, and regional S3 package traffic over AWS networking without an internet path.
Instance and endpoint security groupsExpress approved web ingress, endpoint TLS, and S3 egress as explicit source/destination relationships. Route 53 Resolver DNS Firewall, not a security-group port rule, governs DNS domains when filtering is required.
EC2 instance profileProvides short-lived role credentials for SSM messaging and approved logging only. It contains no application secret or broad managed administrator policy.
Amazon Linux 2023 web hostRuns the reviewed package and configuration with IMDSv2, encrypted storage, private networking, no SSH key pair, and a system service health endpoint.
CloudFormation and CloudTrailPreview and apply infrastructure as a versioned stack, record control-plane actions, detect drift, and preserve failure/rollback events.
  1. Authenticate the operator with a short-lived role, confirm account alias and region without recording the numeric account ID, and capture quotas and existing CIDR ownership.
  2. Review the CloudFormation template, parameters, IAM actions, endpoint set, image parameter, user data, security groups, storage, metadata, tags, deletion behavior, and outputs.
  3. Validate the template, create a change set, wait for completion, and inspect every resource replacement, IAM capability, and policy before execution.
  4. Execute inside an approved window and wait for stack completion; on failure preserve stack events and let CloudFormation roll back rather than patching resources manually.
  5. Prove private routing, no public IP/key pair/SSH, endpoint health, SSM Online status, token-required metadata, volume encryption, web health, logs, and CloudTrail evidence.
  6. Apply updates through new change sets and rehearse stack rollback. Teardown removes termination protection only after retention, dependency, and data checks.

Assumptions

  • The deployment uses one approved AWS account and region, two available zones, non-overlapping RFC1918 CIDRs, and organization guardrails such as SCPs, CloudTrail, Config, and central log retention.
  • AWS CLI v2 runs with a short-lived federated role. No access key, secret key, session token, numeric account ID, or private internal hostname is embedded in the template or tutorial evidence.
  • Amazon Linux 2023 is approved and the selected public SSM parameter is resolved and recorded at change time. Production may replace it with a private golden-image parameter after image scanning and provenance.
  • The web host is reached from an existing approved client/load-balancer security group inside connected private networking. Public exposure requires a separate edge/load-balancer/TLS/WAF design.
  • The region supports the required Systems Manager interface endpoints. SSM Agent is present and running in the approved image, and private DNS/resolver behavior is functional.
  • The stack contains no durable application data. Persistent data belongs on a separately backed-up managed service or volume with retention/deletion policy and recovery tests.

Key concepts

Private subnet
A subnet whose route table has no direct route to an internet gateway. A private address alone is not proof of private routing.
Session Manager
A Systems Manager capability that opens authenticated sessions through the SSM Agent and service control channels without inbound SSH or distributed host keys.
IMDSv2
The token-based EC2 Instance Metadata Service flow. Requiring tokens reduces exposure to unauthenticated metadata requests; hop limit one reduces proxy/container forwarding.
VPC endpoint
Private connectivity to supported AWS services. Interface endpoints use ENIs and security groups; gateway endpoints add service routes for S3 or DynamoDB.
CloudFormation change set
A preview of stack actions including replacement and policy changes. It does not prove workload correctness or absence of runtime side effects.
No wildcard shortcut
Actions, principals, accounts, and resource-capable APIs are enumerated. Resource '*' appears only for specifically enumerated APIs that expose no resource-level authorization and is documented and constrained by other layers.

Before you copy

Values used in this guide

{{awsRegion}}

Approved AWS region selected by the operator profile.

Example: eu-central-1
{{stackName}}

Stable CloudFormation stack name for the private web host.

Example: production-private-web
{{vpcCidr}}

Approved non-overlapping VPC CIDR.

Example: 10.72.0.0/20
{{privateSubnetACidr}}

First private subnet CIDR.

Example: 10.72.0.0/24
{{privateSubnetBCidr}}

Second private subnet CIDR used by endpoints/resilience.

Example: 10.72.1.0/24
{{clientSecurityGroupId}}

Existing security group representing the approved internal load balancer or client tier.

Example: sg-REDACTED_APPROVED_CLIENT
{{s3PrefixListId}}

Regional AWS-managed S3 prefix-list ID resolved before deployment.

Example: pl-REDACTED_REGIONAL_S3
{{instanceType}}

Approved architecture/capacity class supported by the selected image.

Example: t3.small
{{imageParameter}}

SSM parameter resolving to the reviewed Amazon Linux 2023 image.

Example: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
{{releaseVersion}}

Non-secret change identifier used to name reviewed CloudFormation change sets and parameter snapshots.

Example: 2026.07.28-1

Security and production boundaries

  • Do not add port 22, a key pair, public IP, internet gateway, NAT route, or 0.0.0.0/0 egress as a troubleshooting shortcut. Diagnose endpoint, DNS, IAM, and agent state through console/SSM evidence.
  • The instance role contains only enumerated SSM message and logging actions. Where AWS APIs require Resource '*', document the limitation and constrain account, region, endpoint, network, SCP, and monitoring layers.
  • IMDSv2 does not make metadata credentials harmless. Limit the role, prevent untrusted workloads from reaching metadata, use hop limit one, and patch SSRF vulnerabilities.
  • User data is visible to identities with EC2 read permissions and can appear in logs. It must contain no secret, token, certificate private key, account ID, or application credential.
  • CloudFormation parameters and outputs are not secret stores. Use Secrets Manager or Parameter Store with workload identity and avoid exposing dynamic references in logs.
  • Session Manager removes inbound SSH but is privileged remote execution. Require SSO/MFA, least-privilege session permissions, logging, session duration, approval, and alerts.

Stop before continuing if

  • Stop if the caller identity, account alias, region, CIDR ownership, endpoint support, or organization guardrails cannot be verified.
  • Stop if the change set creates a public IP, internet route, port 22, unrestricted ingress/egress, Action/Principal wildcard, unencrypted volume, optional IMDS tokens, or a literal secret.
  • Stop if the selected image or package origin is unverified, the instance role is broader than SSM/logging, or user data contains sensitive material.
  • Stop if any resource replacement affects retained data or a shared network boundary without a tested migration.
  • Stop if SSM does not become Online, metadata v1 still works, EBS is unencrypted, CloudTrail is unavailable, or web health fails.
01

verification

Verify short-lived AWS identity, alias, region, and quotas

read-only

Use a federated CLI profile, confirm role ARN and account alias without copying the numeric account ID into records, set the approved region explicitly, and inspect relevant VPC/endpoint/instance quotas before composing a change.

Why this step matters

The most damaging cloud mistake is a correct command in the wrong account or region. A role ARN and human-readable alias create two identity signals while evidence redacts the numeric account. Short-lived federation limits credential lifetime. Quotas are part of design because endpoint creation can fail after partial stack work. The operator role should create only reviewed network/EC2/IAM resources and should not itself be the instance role.

What to understand

Use AWS_PROFILE and AWS_REGION explicitly in automation rather than relying on a stale default.

Confirm SCP, permission boundary, CloudTrail, Config, and change-window controls.

Record CIDR allocation from the authoritative IPAM source.

System changes

  • Queries identity, alias, configuration, and quotas only.

Syntax explained

--query Arn
Returns the active principal without dumping temporary credentials.
service-quotas get-service-quota
Reads a specific regional/service limit before deployment.
Command
aws sts get-caller-identity --profile production-operator --query Arn --output text && aws iam list-account-aliases --profile production-operator --query 'AccountAliases[0]' --output text && aws configure get region --profile production-operator && aws service-quotas get-service-quota --service-code vpc --quota-code L-29B6F2EB --profile production-operator
Example output / evidence
arn:aws:sts::REDACTED:assumed-role/NetworkChangeOperator/session-redacted
production-platform
eu-central-1
{
  "Quota": {
    "QuotaName": "Interface VPC endpoints per VPC",
    "Value": 50.0
  }
}

Checkpoint: Cloud scope is unambiguous

aws sts get-caller-identity --query Arn --output text && aws ec2 describe-availability-zones --filters Name=state,Values=available --query 'AvailabilityZones[].ZoneName' --output text

Continue whenApproved federated role, production alias, region, two available zones, sufficient quotas, and non-overlapping IPAM allocation.

Stop whenIdentity, alias, region, credential lifetime, guardrails, availability zones, quota, or CIDR differs from approval.

If this step fails

Account alias is empty or unexpected.

Likely causeWrong account or alias not configured.

Safe checks
  • Confirm through organization account inventory.

ResolutionStop and select the correct federated account/profile.

Security notes

  • Redact numeric account IDs and session names in externally shared evidence.

Alternatives

  • Use AWS IAM Identity Center named profiles with enforced MFA.

Stop conditions

  • Stop on any scope ambiguity.
02

verification

Resolve the approved image and regional endpoint dependencies

read-only

Resolve the Amazon Linux 2023 SSM parameter to one AMI, inspect its owner and deprecation state, enumerate Systems Manager endpoint services, and resolve the regional S3 managed prefix list. Record identifiers in protected change evidence, not source code.

Why this step matters

A friendly SSM parameter moves as new images publish. Resolving it at preview time and recording the concrete AMI makes the change auditable. The image owner must be Amazon and state available; organizations should add their own scan/sign pipeline. Endpoint availability differs by region and generation. SSM uses ssm and ssmmessages, with ec2messages still relevant in supported environments. S3 prefix lists support explicit security-group egress for package repositories without arbitrary internet access.

What to understand

Treat AMI ID, owner, architecture, boot mode, deprecation, and root device as one reviewed set.

Use a private golden-image parameter for mature production.

Do not hard-code endpoint service names from another region.

System changes

  • Queries public/service metadata only.

Syntax explained

SSM public AMI parameter
Returns the current AL2023 image ID; record the resolved value.
managed prefix list
AWS-maintained regional destination set for S3 routing/rules.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

AMI_ID=$(aws ssm get-parameter --name '{{imageParameter}}' --query 'Parameter.Value' --output text) && aws ec2 describe-images --image-ids "$AMI_ID" --query 'Images[0].[ImageId,OwnerId,Name,State,DeprecationTime]' --output table && aws ec2 describe-vpc-endpoint-services --query "ServiceNames[?contains(@, '.ssm') || contains(@, '.ssmmessages') || contains(@, '.ec2messages') || contains(@, '.logs')]" --output text && aws ec2 describe-managed-prefix-lists --filters Name=prefix-list-name,Values=com.amazonaws.{{awsRegion}}.s3 --query 'PrefixLists[0].[PrefixListId,PrefixListName]' --output text
Example output / evidence
--------------------------------------------------------------
|                       DescribeImages                       |
+----------------------+----------+---------------------------+
| ami-REDACTED         | amazon   | al2023-ami-2026.07...     |
| available            | None     |                           |
+----------------------+----------+---------------------------+
com.amazonaws.eu-central-1.ssm
com.amazonaws.eu-central-1.ssmmessages
com.amazonaws.eu-central-1.ec2messages
com.amazonaws.eu-central-1.logs
pl-REDACTED  com.amazonaws.eu-central-1.s3

Checkpoint: Image and endpoint dependencies are supported and pinned for review

aws ec2 describe-images --image-ids "$AMI_ID" --query 'Images[0].[State,Public,RootDeviceType,Architecture]' --output text

Continue whenImage is available, EBS-backed, expected architecture, owned by Amazon/approved pipeline, and every endpoint service exists.

Stop whenImage owner/state/architecture is wrong, deprecated unexpectedly, or required endpoint service/prefix list is absent.

If this step fails

Public image parameter resolves to a different AMI during review.

Likely causeA new AL2023 image was published.

Safe checks
  • Compare parameter LastModifiedDate and both image records.

ResolutionRestart image review and change-set generation with one concrete approved AMI.

Security notes

  • AMI IDs are not secrets but can reveal patch cadence and architecture.

Alternatives

  • Use a signed private golden-image SSM parameter.

Stop conditions

  • Stop if image provenance is not accepted.
03

config

Declare private subnets with no internet route

read-only

Create the VPC, two private subnets, DNS support, and dedicated route tables. Do not attach an internet gateway or NAT gateway and do not enable automatic public addressing. Add the S3 gateway endpoint to both route tables.

Why this step matters

Subnet names do not create privacy; routing and interface addressing do. MapPublicIpOnLaunch false prevents automatic public addresses, while absence of internet/NAT routes removes generic outbound internet. Separate route tables make the boundary inspectable. The S3 gateway endpoint adds regional service routes for package content but not arbitrary destinations. DNS support is required for interface endpoint private DNS and normal service discovery.

What to understand

Validate CIDRs against IPAM, peering, transit gateway, on-premises, and service networks.

Network ACLs remain default stateful-compatible; add custom NACLs only with tested return paths.

Tag ownership, environment, data classification, and cost center in production.

System changes

  • Template validation only; later stack execution creates the network.

Syntax explained

MapPublicIpOnLaunch: false
Prevents automatic public IPv4 assignment in the subnet.
Gateway endpoint RouteTableIds
Adds S3 service routes only to the selected private route tables.
File infrastructure/private-web.yml
Configuration
AWSTemplateFormatVersion: "2010-09-09"
Description: Private SSM-managed web host
Parameters:
  VpcCidr: {Type: String}
  PrivateSubnetACidr: {Type: String}
  PrivateSubnetBCidr: {Type: String}
  ClientSecurityGroupId: {Type: AWS::EC2::SecurityGroup::Id}
  S3PrefixListId: {Type: String}
  InstanceType: {Type: String, Default: t3.small}
  ImageId: {Type: AWS::EC2::Image::Id}
Resources:
  Vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags: [{Key: Name, Value: private-web}]
  PrivateSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Ref PrivateSubnetACidr
      AvailabilityZone: !Select [0, !GetAZs ""]
      MapPublicIpOnLaunch: false
  PrivateSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Ref PrivateSubnetBCidr
      AvailabilityZone: !Select [1, !GetAZs ""]
      MapPublicIpOnLaunch: false
  RouteTableA:
    Type: AWS::EC2::RouteTable
    Properties: {VpcId: !Ref Vpc}
  RouteTableB:
    Type: AWS::EC2::RouteTable
    Properties: {VpcId: !Ref Vpc}
  AssociationA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: {SubnetId: !Ref PrivateSubnetA, RouteTableId: !Ref RouteTableA}
  AssociationB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties: {SubnetId: !Ref PrivateSubnetB, RouteTableId: !Ref RouteTableB}
  S3Endpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref Vpc
      ServiceName: !Sub com.amazonaws.${AWS::Region}.s3
      VpcEndpointType: Gateway
      RouteTableIds: [!Ref RouteTableA, !Ref RouteTableB]
Command
aws cloudformation validate-template --template-body file://private-web.yml --query 'Parameters[].ParameterKey' --output text
Example output / evidence
VpcCidr  PrivateSubnetACidr  PrivateSubnetBCidr  ClientSecurityGroupId  S3PrefixListId  InstanceType  ImageId

Checkpoint: Network template has no internet path

grep -nE 'InternetGateway|NatGateway|0\.0\.0\.0/0|MapPublicIpOnLaunch: true' infrastructure/private-web.yml || true

Continue whenNo output and both private subnets have dedicated route tables plus S3 endpoint routes.

Stop whenAny public addressing, internet/NAT route, shared route table, or overlapping CIDR appears.

If this step fails

CloudFormation reports CIDR overlap.

Likely causeVPC or subnet allocation conflicts.

Safe checks
  • Compare IPAM and existing VPC/subnet CIDRs.

ResolutionAllocate new approved non-overlapping ranges; never squeeze or reuse ranges ad hoc.

Security notes

  • Private routes reduce exposure but do not replace SG, IAM, patching, or application auth.

Alternatives

  • Attach to an existing governed private VPC after ownership review.

Stop conditions

  • Stop if private routing cannot be proven.
04

config

Create private service endpoints and explicit network rules

read-only

Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Create interface endpoints in both private subnets with private DNS. The endpoint group accepts TLS only from the instance group. The instance accepts the web port only from the approved client group and sends TLS only to endpoint interfaces and the regional S3 prefix list. AmazonProvidedDNS remains reachable independently of security-group rules; use Route 53 Resolver DNS Firewall when DNS-domain filtering is required.

Why this step matters

Security groups are stateful, so return traffic does not need mirror rules. Referencing groups expresses workload relationships more safely than broad CIDRs. Removing default egress is essential because AWS-created groups otherwise allow all outbound traffic. Endpoint TLS goes only to endpoint ENIs, and S3 TLS goes only to the AWS-managed regional prefix list. AmazonProvidedDNS traffic cannot be blocked or filtered by security groups, so a VPC-CIDR port 53 rule would create false assurance; use Route 53 Resolver DNS Firewall for domain filtering. Current SSM Agent versions prefer ssm and ssmmessages. Add an ec2messages endpoint and the matching minimal IAM actions only when AWS regional endpoint guidance and a tested agent version show that the fallback channel is required.

What to understand

AmazonProvidedDNS bypasses security-group and network-ACL filtering. Verify Route 53 Resolver DNS Firewall policy associations or custom resolver paths separately when DNS control is a requirement.

Use endpoint policies to narrow services where supported; SGs govern network, not API authorization.

The client SG must belong to the same connected security boundary and be separately controlled.

All fragments in this tutorial are parts of one infrastructure/private-web.yml template; merge them under the existing Resources map before validation and change-set creation.

System changes

  • Template review only; stack later creates ENIs and security rules.

Syntax explained

SourceSecurityGroupId
Allows only network interfaces associated with the approved source group.
DestinationPrefixListId
Restricts outbound TLS to AWS-managed regional S3 addresses.
PrivateDnsEnabled
Resolves standard AWS service names to private endpoint addresses in the VPC.
File infrastructure/private-web.yml
Configuration
EndpointSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: TLS from private web instances to AWS endpoints
    VpcId: !Ref Vpc
    SecurityGroupEgress: []
InstanceSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: Private web host explicit paths
    VpcId: !Ref Vpc
    SecurityGroupEgress: []
EndpointTlsIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref EndpointSecurityGroup
    IpProtocol: tcp
    FromPort: 443
    ToPort: 443
    SourceSecurityGroupId: !Ref InstanceSecurityGroup
PrivateWebIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref InstanceSecurityGroup
    IpProtocol: tcp
    FromPort: 8080
    ToPort: 8080
    SourceSecurityGroupId: !Ref ClientSecurityGroupId
EndpointTlsEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !Ref InstanceSecurityGroup
    IpProtocol: tcp
    FromPort: 443
    ToPort: 443
    DestinationSecurityGroupId: !Ref EndpointSecurityGroup
S3TlsEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !Ref InstanceSecurityGroup
    IpProtocol: tcp
    FromPort: 443
    ToPort: 443
    DestinationPrefixListId: !Ref S3PrefixListId
SsmEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref Vpc
    ServiceName: !Sub com.amazonaws.${AWS::Region}.ssm
    VpcEndpointType: Interface
    PrivateDnsEnabled: true
    SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
    SecurityGroupIds: [!Ref EndpointSecurityGroup]
SsmMessagesEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref Vpc
    ServiceName: !Sub com.amazonaws.${AWS::Region}.ssmmessages
    VpcEndpointType: Interface
    PrivateDnsEnabled: true
    SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
    SecurityGroupIds: [!Ref EndpointSecurityGroup]
LogsEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref Vpc
    ServiceName: !Sub com.amazonaws.${AWS::Region}.logs
    VpcEndpointType: Interface
    PrivateDnsEnabled: true
    SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
    SecurityGroupIds: [!Ref EndpointSecurityGroup]
Command
aws cloudformation validate-template --template-body file://private-web.yml >/dev/null && grep -nE 'SecurityGroup(Ingress|Egress)|VPCEndpoint' infrastructure/private-web.yml
Example output / evidence
88:  EndpointSecurityGroup:
101:  InstanceSecurityGroup:
135:  SsmEndpoint:
145:  SsmMessagesEndpoint:
155:  LogsEndpoint:

Checkpoint: Only the documented network relationships exist

grep -nE '0\.0\.0\.0/0|::/0|FromPort: 22|IpProtocol: -1' infrastructure/private-web*.yml || true

Continue whenNo output; web ingress is client-SG-to-instance:8080 and security-group egress is only endpoint TLS and S3 TLS. AmazonProvidedDNS behavior and any Resolver DNS Firewall policy are verified separately.

Stop whenAny unrestricted rule, SSH, public CIDR, default egress, endpoint without private DNS, or unowned client group appears.

If this step fails

Endpoint DNS resolves publicly from the instance.

Likely causePrivate DNS disabled, VPC DNS disabled, or custom resolver bypasses endpoint records.

Safe checks
  • Resolve service names through approved diagnostics and inspect endpoint DNS entries.

ResolutionCorrect VPC/resolver/private-DNS design before starting SSM sessions.

Security notes

  • Endpoint security groups do not replace endpoint policies and IAM.

Alternatives

  • Use centrally shared endpoints through a governed network architecture.

Stop conditions

  • Stop if management needs unrestricted internet egress.
05

config

Create an enumerated SSM and logging instance role

read-only

Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Declare an EC2 trust relationship and list only the current SSM channel, inventory, and log actions required. Some SSM and CloudWatch Logs discovery APIs do not support resource-level ARNs; each Resource '*' is an explicit service limitation paired with enumerated actions, private endpoints, region/account guardrails, and monitoring—not a wildcard shortcut.

Why this step matters

An EC2 role supplies temporary credentials; it should not contain a human login permission or application secrets. The trust policy names only the EC2 service principal. Actions are enumerated. Resource '*' is retained only for listed SSM/channel actions and logs:DescribeLogGroups, whose authorization models have no usable resource ARN; both exceptions must survive Access Analyzer review. Log stream creation and writes are scoped to one log group ARN. The default example omits legacy ec2messages permissions because current SSM Agent versions prefer ssmmessages; add only the exact fallback actions after verifying that the selected region and agent require them. Organization SCPs, endpoint policy, region controls, instance profile association, and CloudTrail further limit and observe use.

What to understand

Do not attach AdministratorAccess or a broad convenience policy.

Separate application data permissions into another narrowly reviewed statement/role only when required.

Validate both identity and trust policies; Access Analyzer findings require disposition, not blind suppression.

System changes

  • Validates a policy document only; stack later creates role/profile.

Syntax explained

Principal ec2.amazonaws.com
Allows the EC2 service to deliver role credentials to associated instances.
enumerated Action list
Prevents service-wide or account-wide action wildcards.
LogGroup.Arn:*
Scopes log streams/events beneath one controlled group.
File infrastructure/private-web.yml
Configuration
SessionLogGroup:
  Type: AWS::Logs::LogGroup
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    LogGroupName: !Sub /aws/ssm/${AWS::StackName}
    RetentionInDays: 90
InstanceRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal: {Service: ec2.amazonaws.com}
          Action: sts:AssumeRole
    Policies:
      - PolicyName: SsmMessagingAndInventory
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: SsmControlChannelsNoResourceArn
              Effect: Allow
              Action:
                - ssmmessages:CreateControlChannel
                - ssmmessages:CreateDataChannel
                - ssmmessages:OpenControlChannel
                - ssmmessages:OpenDataChannel
                - ssm:UpdateInstanceInformation
              Resource: "*"
            - Sid: DiscoverConfiguredSessionLogGroup
              Effect: Allow
              Action: logs:DescribeLogGroups
              Resource: "*"
            - Sid: SessionLogsOnly
              Effect: Allow
              Action:
                - logs:CreateLogStream
                - logs:DescribeLogStreams
                - logs:PutLogEvents
              Resource: !Sub ${SessionLogGroup.Arn}:*
InstanceProfile:
  Type: AWS::IAM::InstanceProfile
  Properties:
    Roles: [!Ref InstanceRole]
Command
aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://instance-policy.json --query 'findings[].[findingType,issueCode]' --output table
Example output / evidence
---------------------------------
|        ValidatePolicy         |
+--------------+----------------+
| findingType  | issueCode      |
+--------------+----------------+
| SUGGESTION   | CREATE_SLR_WITH_NOT_RESOURCE |
+--------------+----------------+

Checkpoint: Role has no wildcard action/principal and every resource exception is justified

aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://instance-policy.json

Continue whenNo ERROR/SECURITY_WARNING remains; SSM channel and logs:DescribeLogGroups Resource-star limitations are documented; log stream writes are scoped to the exact log group.

Stop whenAction/Principal wildcard, broad managed policy, unrelated service, write-capable application permission, or unexplained analyzer finding appears.

If this step fails

Analyzer warns about Resource '*'.

Likely causeThe enumerated SSM API does not support resource-level permissions or policy is broader than needed.

Safe checks
  • Check AWS Service Authorization reference for each action.

ResolutionKeep only unavoidable enumerated actions and add conditions/guardrails where supported; remove every resource-capable wildcard.

Security notes

  • Metadata access exposes role credentials; least privilege remains mandatory even with IMDSv2.

Alternatives

  • Use the current AWS-managed SSM core policy only after reviewing its exact version and broader statements; custom policy is easier to audit here.

Stop conditions

  • Stop if SSM cannot operate without unrelated authority.
06

config

Require IMDSv2, encrypted gp3, no key pair, and protected lifecycle

read-only

Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Launch one private Amazon Linux 2023 instance without a public IP or key pair. Require metadata tokens, hop limit one, disabled metadata tags, encrypted gp3 storage, detailed monitoring, termination protection, and minimal non-secret bootstrap.

Why this step matters

HttpTokens required rejects IMDSv1; hop limit one reduces forwarding beyond the instance network boundary. Instance metadata tags are disabled to avoid exposing tag content locally. Encrypted gp3 protects the root volume at rest; account default encryption is helpful but the template states intent. No KeyName means no SSH key is installed, and the SG has no SSH ingress. Termination protection prevents casual API deletion but must be deliberately removed for replacement/teardown. User data is intentionally secret-free and fails on error.

What to understand

Prefer a golden image so bootstrap does not depend on package repositories.

Use TLS at an internal load balancer or configure a private certificate through workload identity; never embed a key in user data.

Size storage and instance class through measured workload requirements.

System changes

  • Template review only; stack later launches and bootstraps the instance.

Syntax explained

HttpTokens: required
Requires a session token for all metadata requests.
HttpPutResponseHopLimit: 1
Restricts metadata token responses to one network hop.
DisableApiTermination: true
Blocks termination until explicitly disabled.
File infrastructure/private-web.yml
Configuration
WebInstance:
  Type: AWS::EC2::Instance
  Properties:
    ImageId: !Ref ImageId
    InstanceType: !Ref InstanceType
    IamInstanceProfile: {Name: !Ref InstanceProfile}
    SubnetId: !Ref PrivateSubnetA
    SecurityGroupIds: [!Ref InstanceSecurityGroup]
    DisableApiTermination: true
    Monitoring: true
    MetadataOptions:
      HttpEndpoint: enabled
      HttpTokens: required
      HttpPutResponseHopLimit: 1
      InstanceMetadataTags: disabled
    BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          Encrypted: true
          VolumeType: gp3
          VolumeSize: 16
          DeleteOnTermination: true
    Tags:
      - {Key: Name, Value: private-web}
      - {Key: ManagedBy, Value: cloudformation}
    UserData:
      Fn::Base64: |
        #!/bin/bash
        set -euo pipefail
        dnf -y install nginx
        cat >/etc/nginx/conf.d/private-web.conf <<'CONF'
        server {
          listen 8080;
          server_name _;
          server_tokens off;
          location = /healthz { return 200 "healthy
"; }
          location / { return 404; }
        }
        CONF
        nginx -t
        systemctl enable --now nginx
Command
aws cloudformation validate-template --template-body file://private-web.yml >/dev/null && grep -nE 'MetadataOptions|Encrypted|AssociatePublicIpAddress|DisableApiTermination|KeyName' infrastructure/private-web.yml
Example output / evidence
214:      DisableApiTermination: true
221:          AssociatePublicIpAddress: false
229:      MetadataOptions:
235:          Encrypted: true

Checkpoint: Launch declaration has every hardening invariant

grep -nE 'HttpTokens: required|HttpPutResponseHopLimit: 1|Encrypted: true|AssociatePublicIpAddress: false|DisableApiTermination: true' infrastructure/private-web*.yml

Continue whenAll invariants are present, no KeyName/public IP/secret exists, and image plus bootstrap are reviewed.

Stop whenMetadata tokens optional, storage plaintext, public addressing, SSH key, secret user data, mutable image, or termination protection absent.

If this step fails

User data finishes partially.

Likely causePackage/DNS/S3 endpoint failure or nginx validation error.

Safe checks
  • Read cloud-init output and systemd status through SSM/serial console.

ResolutionCorrect template/image and replace through a change set; do not manually complete hidden drift.

Security notes

  • EC2 read-user-data permission exposes bootstrap; keep it non-sensitive.

Alternatives

  • Use a scanned golden AMI with no bootstrap package installation.

Stop conditions

  • Stop if bootstrap cannot complete without broad egress or secrets.
07

command

Create a CloudFormation change set and inspect every action

read-only

Validate the final template, create a named change set with explicit parameters and IAM capability, wait for creation, then inspect action, replacement, scope, details, and policy changes. Do not execute from a pipeline that hides the change set.

Why this step matters

CloudFormation validation checks template structure but not architectural safety. A change set reveals concrete resource actions and whether an update replaces a resource. CAPABILITY_NAMED_IAM is explicit acknowledgement that the stack creates named IAM resources; it is not authorization to skip policy review. Parameter files can expose topology and must not contain secrets. CREATE versus UPDATE type must match stack state.

What to understand

Review all parameters, tags, resolved AMI, endpoint services, SG rules, role policies, user data, outputs, deletion/update policies, and replacements.

Run Access Analyzer on extracted policies and organization policy-as-code before execution.

Delete stale unexecuted change sets to prevent accidental later execution.

System changes

  • Creates a CloudFormation change-set preview but no stack resources.

Syntax explained

CAPABILITY_NAMED_IAM
Acknowledges creation/modification of explicitly named IAM resources.
Replacement
Shows whether update requires a new physical resource.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

aws cloudformation validate-template --template-body file://infrastructure/private-web.yml && aws cloudformation create-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --change-set-type CREATE --template-body file://infrastructure/private-web.yml --parameters file://infrastructure/parameters.json --capabilities CAPABILITY_NAMED_IAM --query Id --output text && aws cloudformation wait change-set-create-complete --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} && aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --query 'Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement,Scope]' --output table
Example output / evidence
Template validation: parameters accepted
arn:aws:cloudformation:eu-central-1:REDACTED:changeSet/reviewed-2026.07.28-1/REDACTED
--------------------------------------------------------------------------------
|                              DescribeChangeSet                               |
+--------+------------------------+------------------------------+-------+-------+
| Add    | Vpc                    | AWS::EC2::VPC                | False | None  |
| Add    | SsmEndpoint            | AWS::EC2::VPCEndpoint        | False | None  |
| Add    | InstanceRole           | AWS::IAM::Role               | False | None  |
| Add    | WebInstance            | AWS::EC2::Instance           | False | None  |

Checkpoint: Change set contains only accepted resources and no hidden replacement

aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --output json > /secure-staging/change-set.json && sha256sum /secure-staging/change-set.json

Continue whenEvery action, IAM statement, network rule, image, storage property, and replacement is accepted and snapshot digest recorded.

Stop whenUnexpected delete/replace, wildcard, public route/address, plaintext storage, broad role, secret parameter, or unreviewed resource appears.

If this step fails

Change set status is FAILED with no changes.

Likely causeTemplate equals live stack or wrong CREATE/UPDATE type.

Safe checks
  • Read StatusReason and stack existence.

ResolutionTreat no changes as evidence; do not force an unrelated mutation.

Security notes

  • Change-set JSON can reveal internal network and IAM design; protect it.

Alternatives

  • Use CloudFormation StackSets with delegated admin after the same per-account preview controls.

Stop conditions

  • Stop until a human reviewer approves exact physical actions.
08

command

Execute the approved change set and preserve stack events

caution

Execute only the reviewed change-set ID, wait for stack completion, and export stack events and outputs with account identifiers redacted. Let CloudFormation roll back on failure; do not disable rollback for production.

Why this step matters

The change-set name and stored digest bind execution to review. CloudFormation orders dependencies and records each resource event. A stack CREATE_COMPLETE means infrastructure APIs succeeded, not that SSM, metadata, web health, logging, and security invariants work. Rollback on failure should remain enabled so partially created resources are removed where possible, but event evidence must be captured first.

What to understand

Monitor the first failure event, not only final stack status.

Avoid manual fixes because they create drift and may break rollback.

Redact account IDs and sensitive internal outputs from external evidence.

System changes

  • Creates VPC, subnets, endpoints, SGs, IAM role/profile, logs, and EC2 instance.

Syntax explained

execute-change-set
Applies the previously generated concrete change plan.
wait stack-create-complete
Returns only after terminal create success/failure.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

aws cloudformation execute-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} && aws cloudformation wait stack-create-complete --stack-name {{stackName}} && aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].[StackStatus,Outputs]' --output json && aws cloudformation describe-stack-events --stack-name {{stackName}} --max-items 30 --output table
Example output / evidence
[
  "CREATE_COMPLETE",
  [
    {"OutputKey": "InstanceId", "OutputValue": "i-REDACTED"},
    {"OutputKey": "VpcId", "OutputValue": "vpc-REDACTED"}
  ]
]
CREATE_COMPLETE  AWS::CloudFormation::Stack     production-private-web
CREATE_COMPLETE  AWS::EC2::Instance            WebInstance
CREATE_COMPLETE  AWS::EC2::VPCEndpoint         SsmEndpoint

Checkpoint: Stack API operation completes without manual drift

aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output text

Continue whenCREATE_COMPLETE and no manual resource edit or unresolved warning event.

Stop whenRollback begins, resource fails, endpoint/role differs, or anyone proposes console patching outside the template.

If this step fails

Stack rolls back after instance creation.

Likely causeEndpoint, IAM, subnet, bootstrap, or wait-condition dependency failed.

Safe checks
  • Preserve ordered stack events and first failure reason.

ResolutionLet rollback complete, correct source template, create a new change set.

Security notes

  • CloudFormation events can include property identifiers; protect exports.

Alternatives

  • Execute through a protected CI role after human change-set approval.

Stop conditions

  • Stop post-deploy acceptance until stack is stable.
09

verification

Prove no public IP, internet route, SSH, or unrestricted rule exists

read-only

Describe the instance interfaces, subnet public-address setting, route tables, and all SG rules by rule ID. Verify no internet gateway/NAT route, public address, key pair, port 22, or all-address/all-protocol rule exists.

Why this step matters

Template intent can be changed by defaults, drift, or another tool. Runtime inspection proves actual interfaces, routes, and SG rules. KeyName None confirms no EC2 key pair was injected; it does not prove sshd package absence, so port/process checks follow through SSM. Rule IDs support exact revocation and CloudTrail correlation.

What to understand

Inspect every ENI, not only primary interface.

Check subnet MapPublicIpOnLaunch independently.

Run CloudFormation drift after runtime inspection.

System changes

  • Reads EC2/VPC state only.

Syntax explained

PublicIpAddress None
No public IPv4 is associated with the instance.
route target vpce
S3 service traffic uses the gateway endpoint rather than internet.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name {{stackName}} --query "Stacks[0].Outputs[?OutputKey=='InstanceId'].OutputValue" --output text) && aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[PrivateIpAddress,PublicIpAddress,KeyName,MetadataOptions.HttpTokens]' --output table && aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].SubnetId' --output text) --query 'RouteTables[0].Routes' --output table && aws ec2 describe-security-group-rules --filters Name=group-id,Values=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupId' --output text) --output table
Example output / evidence
------------------------------------------------
|               DescribeInstances              |
+-------------+--------+-------+----------------+
| 10.72.0.34  | None   | None  | required       |
+-------------+--------+-------+----------------+
| Destination | Target             | State       |
| 10.72.0.0/20| local              | active      |
| pl-REDACTED | vpce-REDACTED      | active      |
No rules matching port 22, 0.0.0.0/0, ::/0, or protocol -1

Checkpoint: Runtime network is private and explicit

aws cloudformation detect-stack-drift --stack-name {{stackName}} --query StackDriftDetectionId --output text

Continue whenNo public address/key, no default internet route, no SSH/broad rules, and later drift result is IN_SYNC.

Stop whenAny exposure, unknown ENI/route/rule, or drift exists.

If this step fails

Unexpected SG rule exists outside template.

Likely causeManual change or another automation owner.

Safe checks
  • Correlate rule ID with CloudTrail and stack drift.

ResolutionTreat as incident, revoke under approval, and restore single ownership.

Security notes

  • Network identifiers are sensitive operational metadata.

Alternatives

  • Use AWS Config rules for continuous public-exposure detection.

Stop conditions

  • Stop acceptance on any unreviewed reachability.
10

verification

Verify SSM Online status and an attributable no-SSH session

caution

Wait for the instance to register as a managed node, inspect agent/platform data, start one approved Session Manager shell session, run non-mutating identity and service checks, then terminate it and correlate the CloudTrail event and compatible session-content log. CloudTrail records session lifecycle, but SSH and port-forwarding sessions through Session Manager do not support command-content logging.

Why this step matters

Online status proves the agent, role, DNS, endpoints, and service control channels are functioning. Session Manager authenticates the operator through IAM and avoids inbound SSH keys, but it still grants interactive host access and often sudo through ssm-user configuration. Session policy, logging, MFA, duration, approvals, and alerting must constrain it. Use the session only for read-only acceptance; durable configuration belongs in the image/template.

What to understand

Configure Session Manager preferences to CloudWatch Logs/S3 with encryption before routine use.

Restrict start-session to tagged instances and approved documents.

Alert on StartSession, ResumeSession, and TerminateSession.

Do not describe SSH or port-forwarding sessions as content-audited: Session Manager cannot log their commands or tunneled traffic. Use only session lifecycle events for those modes.

System changes

  • Starts and terminates an attributable interactive shell session; commands shown are read-only. Compatible content logging is verified separately from CloudTrail lifecycle logging.

Syntax explained

describe-instance-information
Shows managed-node connectivity and agent/platform metadata.
start-session --target
Opens a privileged management channel; requires explicit IAM authorization.
Command
aws ssm describe-instance-information --filters Key=InstanceIds,Values="$INSTANCE_ID" --query 'InstanceInformationList[0].[PingStatus,PlatformName,PlatformVersion,AgentVersion]' --output table && aws ssm start-session --target "$INSTANCE_ID"
Example output / evidence
-------------------------------------------------
|          DescribeInstanceInformation          |
+--------+------------------+----------+---------+
| Online | Amazon Linux     | 2023     | 3.3.x   |
+--------+------------------+----------+---------+
Starting session with SessionId: operator-REDACTED
sh-5.2$ id
uid=1000(ssm-user) gid=1000(ssm-user) groups=1000(ssm-user)
sh-5.2$ systemctl is-active nginx
active
sh-5.2$ exit
Exiting session.

Checkpoint: Management works without inbound SSH and produces audit evidence

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=StartSession --max-results 5 --query 'Events[0].[EventName,Username]' --output table

Continue whenNode Online, session identity approved, no port 22/key pair, session terminated, CloudTrail lifecycle event present, and content log present only for the compatible shell-session mode tested here.

Stop whenLogging missing, session identity/permissions broader than policy, node Offline, or anyone proposes SSH fallback.

If this step fails

StartSession denied while node is Online.

Likely causeOperator IAM lacks target/document/session permissions or conditions mismatch tags.

Safe checks
  • Inspect encoded authorization message and simulate operator policy.

ResolutionGrant only the exact session document/target conditions; do not broaden instance role.

Security notes

  • Interactive sessions are privileged and may expose host data; minimize and audit.

Alternatives

  • Use Run Command with an allowlisted read-only document for routine diagnostics.

Stop conditions

  • Stop if session cannot be logged and attributed.
11

verification

Test IMDSv2 enforcement, EBS encryption, listeners, and bootstrap

read-only

Through the approved, attributable shell session, require an unauthenticated metadata request to fail, obtain a short-lived token for a permitted request, inspect nginx, listeners, package signature state, and cloud-init. From the control plane, verify every attached EBS volume is encrypted.

Why this step matters

Control-plane metadata options and an on-host negative probe together prove enforcement. A 401 without token is expected. Token lifetime should be short for diagnostics and token output never logged. Every attached volume, including later data volumes, must be checked. Nginx configuration validation and listener inspection prove the service uses the intended port, while cloud-init status and package verification establish bootstrap completion.

What to understand

Never print role credentials from metadata.

Check unexpected listeners and sshd state; no ingress rule is not permission to ignore host services.

Verify KMS key policy when a customer-managed key replaces aws/ebs.

System changes

  • Performs metadata/storage/service reads only.

Syntax explained

HttpTokens required
Rejects metadata requests that omit a valid PUT token.
hop limit 1
Limits token response forwarding beyond the host network context.
Command
aws ec2 describe-volumes --filters Name=attachment.instance-id,Values="$INSTANCE_ID" --query 'Volumes[].{Id:VolumeId,Encrypted:Encrypted,Type:VolumeType,KmsKeyId:KmsKeyId}' --output table && aws ec2 describe-instance-metadata-options --instance-ids "$INSTANCE_ID" --query 'InstanceMetadataOptions[0].[HttpTokens,HttpPutResponseHopLimit,InstanceMetadataTags,State]' --output table
Example output / evidence
--------------------------------------------------------
|                    DescribeVolumes                   |
+--------------+-----------+------+--------------------+
| vol-REDACTED | True      | gp3  | alias/aws/ebs ...  |
+--------------+-----------+------+--------------------+
------------------------------------------------
|       DescribeInstanceMetadataOptions        |
+----------+--------+----------+----------------+
| required | 1      | disabled | applied        |
+----------+--------+----------+----------------+
IMDSv1 probe: HTTP 401
nginx: configuration file /etc/nginx/nginx.conf test is successful
LISTEN 0 511 0.0.0.0:8080 users:(("nginx",pid=1284,fd=6))

Checkpoint: Runtime matches metadata, disk, and service hardening

aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[State.Name,MetadataOptions.HttpTokens,MetadataOptions.HttpPutResponseHopLimit]' --output text

Continue whenInstance running, tokens required, hop limit one, every EBS encrypted, IMDSv1 401, expected listener only, bootstrap and nginx validation successful.

Stop whenMetadata v1 succeeds, credential output appears, any volume plaintext, unexpected listener/package, or cloud-init failed.

If this step fails

Volume encryption true but unexpected KMS key.

Likely causeAccount default key, launch template, or copied AMI snapshot uses another key.

Safe checks
  • Inspect volume, snapshot, default encryption, and key policy metadata.

ResolutionValidate key ownership/policy or replace through encrypted snapshot migration.

Security notes

  • Never retrieve /latest/meta-data/iam/security-credentials role values for testing.

Alternatives

  • Disable metadata endpoint entirely if the workload and agent do not require it.

Stop conditions

  • Stop on any failed hardening invariant.
12

verification

Verify the web path from the approved client tier

read-only

Resolve the instance private address through stack outputs or service discovery and run the health request from a host associated with the approved client security group. Confirm that an unapproved test source cannot connect and that the application exposes only the intended health/404 behavior.

Why this step matters

Local health proves process state but not network policy. A request from the approved client tier exercises routing, SG source relationship, listener, and application. A negative test from an unapproved SG proves ingress is not accidentally broader. The root endpoint returns 404 to minimize accidental content; a real application would add authenticated routes and TLS termination.

What to understand

Use VPC Flow Logs to correlate accepted and rejected paths.

Do not expose a public test endpoint to simplify validation.

Test availability over both subnets/endpoints in the multi-instance next stage.

System changes

  • Sends safe HTTP GET requests only.

Syntax explained

--connect-timeout 3 --max-time 5
Bounds a failed network/application check.
--fail
Returns non-zero for HTTP errors on the health path.
Command
curl --fail --silent --show-error --connect-timeout 3 --max-time 5 http://PRIVATE_ADDRESS_REDACTED:8080/healthz && curl --silent --output /dev/null --write-out '%{http_code}\n' http://PRIVATE_ADDRESS_REDACTED:8080/
Example output / evidence
healthy
404

Checkpoint: Only approved private clients reach the healthy service

aws ec2 describe-security-group-rules --filters Name=group-id,Values="$INSTANCE_SG" --query 'SecurityGroupRules[?IsEgress==`false`].[FromPort,ToPort,ReferencedGroupInfo.GroupId]' --output table

Continue whenApproved client gets 200 healthy, root behavior matches design, unapproved source is denied, and only client SG port 8080 ingress exists.

Stop whenPublic/unapproved source connects, health differs, listener exposes extra content, or logs contain private data.

If this step fails

Approved client times out but SSM local curl succeeds.

Likely causeSG reference, routing, NACL, client subnet, or service bind path is wrong.

Safe checks
  • Inspect flow logs, routes, NACLs, SG rule IDs, and listener address.

ResolutionCorrect only the intended private path through a change set.

Security notes

  • Health endpoints should reveal minimal status, not version, dependency, or secret details.

Alternatives

  • Use an internal ALB target health check and authenticated synthetic monitor.

Stop conditions

  • Stop acceptance until positive and negative network tests pass.
13

verification

Correlate CloudTrail, stack drift, logs, and security findings

read-only

Confirm CloudTrail records stack, EC2, IAM, endpoint, SG, and Session Manager actions; run stack drift; verify session/web logs and create alerts for public exposure, metadata weakening, SG changes, role changes, SSM Offline, disk/CPU, and service health.

Why this step matters

CloudFormation ownership is credible only while drift remains detected and remediated. CloudTrail records control-plane calls but data events and session content require separate configuration. Monitoring must cover both availability and hardening regression: public addresses/routes, SG rule changes, metadata options, volume encryption, role/policy changes, endpoint state, agent connectivity, service errors, and instance resource pressure. An alert without owner and runbook is not a control.

What to understand

Verify organization trail coverage, region, encryption, retention, validation, and delivery alarms.

Enable AWS Config/Access Analyzer/Security Hub according to account standards.

Store session logs in a separate protected log account where possible.

System changes

  • Starts read-only drift detection and audit queries.

Syntax explained

detect-stack-drift
Compares supported live resource properties with stack expectations.
lookup-events
Queries recent management events; not a replacement for retained trail logs.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

DETECTION_ID=$(aws cloudformation detect-stack-drift --stack-name {{stackName}} --query StackDriftDetectionId --output text) && aws cloudformation wait stack-drift-detection-complete --stack-drift-detection-id "$DETECTION_ID" && aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id "$DETECTION_ID" --query '[DetectionStatus,StackDriftStatus,DriftedStackResourceCount]' --output table && aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue={{stackName}} --max-results 10 --output table
Example output / evidence
---------------------------------------
| DescribeStackDriftDetectionStatus |
+-----------+----------+-------------+
| DETECTION_COMPLETE | IN_SYNC | 0   |
+-----------+----------+-------------+
LookupEvents: CreateChangeSet, ExecuteChangeSet, CreateStack recorded

Checkpoint: Ownership and audit remain observable

aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].[StackStatus,EnableTerminationProtection]' --output table

Continue whenStack IN_SYNC, termination protection policy accepted, CloudTrail session-lifecycle logs present, compatible shell-session content logging tested, alerts exercised, and owners/runbooks assigned.

Stop whenDrift, missing trail/session evidence, disabled alert, or uncontrolled policy/network change remains.

If this step fails

Stack reports NOT_CHECKED or UNKNOWN drift.

Likely causeUnsupported resources, detection failure, or insufficient permissions.

Safe checks
  • Inspect detection status reason and per-resource drift support.

ResolutionAdd direct invariant checks for unsupported properties and repair detection permissions.

Security notes

  • Central logs contain operator identity and host activity; restrict and retain appropriately.

Alternatives

  • Use AWS Config conformance packs plus custom controls for continuous validation.

Stop conditions

  • Stop release closure until audit and monitoring gaps have owners.
14

warning

Rehearse safe updates, CloudFormation rollback, and protected teardown

danger

Create an update change set for a reversible web configuration/image change, reject unexplained replacements, execute, and verify. On failure let CloudFormation roll back and inspect first-cause events. Teardown requires data retention, dependency checks, and explicit removal of instance termination protection before stack deletion.

Why this step matters

Updates can replace an instance or endpoint even when a template diff looks small. The change set's Replacement property and application data model decide safety. CloudFormation rollback restores declared infrastructure where possible, but it cannot reverse external side effects or data migrations. Termination protection intentionally complicates delete/replace and must be toggled only under approved recovery/teardown. A single host causes downtime during replacement; production should evolve to a multi-instance immutable rollout.

What to understand

Use a harmless reversible update and record before/after AMI, package, metadata, SG, role, and health state.

Never skip resources in continue-update-rollback without understanding resulting drift.

Before delete, export evidence, revoke sessions, detach retained data, and verify no endpoint/network dependency is shared.

System changes

  • Creates an update preview; executing later may replace or reconfigure resources.

Syntax explained

change-set-type UPDATE
Calculates changes against the existing stack.
Replacement Conditional
Replacement depends on actual property delta and needs detailed review.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

aws cloudformation create-change-set --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} --change-set-type UPDATE --template-body file://infrastructure/private-web.yml --parameters file://infrastructure/rollback-drill-parameters.json --capabilities CAPABILITY_NAMED_IAM --query Id --output text && aws cloudformation wait change-set-create-complete --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} && aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} --query 'Changes[].ResourceChange.[Action,LogicalResourceId,Replacement]' --output table
Example output / evidence
arn:aws:cloudformation:eu-central-1:REDACTED:changeSet/rollback-drill/REDACTED
---------------------------------------------------------
|                   DescribeChangeSet                   |
+--------+------------------------+----------------------+
| Modify | WebInstance            | Conditional          |
+--------+------------------------+----------------------+

Checkpoint: Update and rollback preserve private/hardening invariants

aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output text

Continue whenStack returns UPDATE_COMPLETE or UPDATE_ROLLBACK_COMPLETE, then all SSM/network/IMDS/EBS/web/audit checks pass.

Stop whenReplacement is unexplained, data retained on root disk, rollback cannot restore, termination protection bypass is improvised, or hardening regresses.

If this step fails

Delete stack hangs on protected instance.

Likely causeDisableApiTermination remains true by design.

Safe checks
  • Confirm stack delete intent, retention evidence, and physical instance identity.

ResolutionDisable termination protection through an approved update/API action, record it, then retry controlled deletion.

Security notes

  • Teardown must revoke instance/session authority and inspect retained snapshots/logs.

Alternatives

  • Use blue/green ASG replacement and traffic shift rather than in-place update.

Stop conditions

  • Stop any rollback/delete that could remove unbacked data or shared network dependencies.

Finish line

Verification checklist

No public management exposureaws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[PublicIpAddress,KeyName]' --output textBoth values are None; no port 22, internet route, or broad network rule exists.
SSM private managementaws ssm describe-instance-information --filters Key=InstanceIds,Values="$INSTANCE_ID" --query 'InstanceInformationList[0].PingStatus' --output textInstance is Online through private endpoints; session lifecycle is attributable in CloudTrail, and compatible shell-session content logging is verified without implying coverage for SSH or port forwarding.
Metadata and storageaws ec2 describe-instance-metadata-options --instance-ids "$INSTANCE_ID" && aws ec2 describe-volumes --filters Name=attachment.instance-id,Values="$INSTANCE_ID"IMDS tokens required, hop limit one, tags disabled, and every attached EBS volume encrypted.
Private web healthcurl --fail --connect-timeout 3 --max-time 5 http://PRIVATE_ADDRESS_REDACTED:8080/healthzApproved client tier receives healthy while unapproved sources are denied.
Stack ownership and auditaws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output textStack is complete/in sync, CloudTrail and session evidence exist, and alarms are active.

Recovery guidance

Common problems and safe checks

The instance never becomes a Systems Manager managed node.

Likely causeSSM Agent is absent/stopped, role policy is incomplete, endpoint DNS/security groups are wrong, time is skewed, or the subnet cannot reach required services.

Safe checks
  • Inspect stack events, endpoint state/private DNS, instance profile association, and SSM inventory.
  • Use EC2 serial console only under approved break-glass access to read agent logs.

ResolutionCorrect the image, explicit policy, DNS, endpoint, or security-group configuration through a new change set; do not open SSH or internet egress.

Interface endpoint creation fails.

Likely causeService unavailable in region/AZ, private DNS conflict, subnet address exhaustion, quota, or duplicate endpoint exists.

Safe checks
  • Describe endpoint services, subnets, existing endpoints, DNS settings, and quotas.

ResolutionChoose supported subnets/services or adopt the existing reviewed endpoint through a separate architecture change.

dnf cannot install the web package.

Likely causeS3 gateway route/egress, DNS, repository metadata, image release, or package signature is unavailable.

Safe checks
  • Read bounded cloud-init/dnf logs through SSM or console and inspect S3 endpoint routes.

ResolutionRepair private package connectivity or use a pre-baked golden image; do not add unrestricted NAT/internet egress.

IMDS request without a token succeeds.

Likely causeLaunch template MetadataOptions did not apply, instance was replaced from another template, or HttpTokens remains optional.

Safe checks
  • Describe instance metadata options and launch template version.

ResolutionUpdate through a reviewed change set to HttpTokens required and replace/restart only as documented.

Instance has a public IPv4 address.

Likely causeSubnet mapPublicIpOnLaunch, network interface AssociatePublicIpAddress, or manual attachment overrides private design.

Safe checks
  • Describe subnet and instance network interfaces and stack drift.

ResolutionStop service acceptance, remove exposure through CloudFormation, rotate any credentials exposed during the interval, and investigate drift.

Web health works locally but not from the client tier.

Likely causeSource security group, port, routing, NACL, listener, DNS, or service bind address differs.

Safe checks
  • Verify service listener, instance/client SG references, route tables, NACLs, and VPC flow logs.

ResolutionCorrect the narrow intended path; never replace source-SG ingress with broad CIDR access.

CloudFormation change set shows instance replacement.

Likely causeImmutable subnet, image, storage, or launch property changed.

Safe checks
  • Inspect Replacement and Scope for the exact resource, plus data/availability dependencies.

ResolutionUse a tested replacement/traffic migration or reject the change; never execute an unexplained replacement.

Stack enters UPDATE_ROLLBACK_FAILED.

Likely causeA resource cannot restore, was manually changed, dependency is unavailable, or termination protection blocks replacement/deletion.

Safe checks
  • Preserve stack events and identify the first failed logical resource.

ResolutionUse continue-update-rollback with explicit resources only after recovery review; avoid manual divergence.

Session starts but no audit log appears.

Likely causeSession Manager preferences, log group/KMS permission, region, or CloudWatch endpoint is missing.

Safe checks
  • Inspect Session Manager preferences, explicit IAM actions, endpoint, log stream, and CloudTrail StartSession event.

ResolutionBlock routine sessions until logging is repaired, except documented emergency access with compensating evidence.

EBS volume reports unencrypted.

Likely causeBlockDeviceMappings omitted Encrypted, an attached data volume is outside the stack, or account default encryption is off.

Safe checks
  • Describe every attached volume and launch template block mapping.

ResolutionSnapshot/copy with encryption and replace through a tested migration; do not copy sensitive data to plaintext storage.

Security group drift adds broad egress.

Likely causeManual rule, another stack/tool, or default SG behavior changed the resource.

Safe checks
  • Run stack drift, describe rules with identifiers, and correlate CloudTrail authorization events.

ResolutionRevoke unauthorized rule under incident control, rotate exposed credentials if needed, and restore stack ownership.

Web service restarts repeatedly after boot.

Likely causeUser data/package/config, port conflict, permissions, resource pressure, or dependency failure.

Safe checks
  • Read bounded systemd, nginx, cloud-init, and kernel logs through audited SSM.

ResolutionReplace with a corrected reviewed image/template or roll back the stack update; do not edit production manually without recording drift.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Use an Auto Scaling Group and internal Application Load Balancer across both private subnets for production availability; retain the same SSM, metadata, storage, network, and preview controls.
  • Use a golden AMI pipeline to preinstall and scan the web package, avoiding runtime package download and making instance replacement deterministic.
  • Use ECS/Fargate or another managed runtime when host administration is not a product requirement.

Operate it safely

  • Replace the single instance with a multi-AZ Auto Scaling Group, internal load balancer health checks, rolling instance refresh, and capacity alarms.
  • Add VPC Flow Logs, CloudWatch agent/metrics, AWS Config managed/custom rules, Inspector, image scanning, and Security Hub findings.
  • Move web artifacts into a signed golden-image pipeline and verify image provenance before launch.
  • Add a dedicated edge design with TLS, WAF, DDoS protection, authenticated origin, and no direct instance exposure if public service is required.
  • Test account/region disaster recovery and CloudFormation restore from a protected repository.

Recovery

Rollback

Use a reviewed CloudFormation update change set to restore the previous image/template. Preserve first-cause events and let automatic rollback complete. The host is replaceable and contains no durable data. If teardown is required, verify retention/dependencies, disable instance API termination protection deliberately, delete through the stack, and confirm endpoints, ENIs, roles, volumes, and logs follow their retention policy.

  1. Freeze further change sets and capture stack/change-set IDs, events, template digest, resolved AMI, instance/network/role state, SSM status, health, and CloudTrail.
  2. Keep the host unreachable from clients if health or security fails; do not open SSH/public egress for repair.
  3. Let CREATE/UPDATE rollback complete and identify the first failed logical resource.
  4. If automatic rollback succeeds, rerun every private-network, SSM, IMDSv2, EBS, web, log, and drift check.
  5. If rollback fails, use continue-update-rollback only with reviewed resource skips and a plan to remove resulting drift.
  6. For a forward/previous-image update, create and review a new change set and reject unexplained replacements or policy expansion.
  7. Before deletion, export evidence, confirm no durable data/shared endpoint, terminate sessions, and satisfy retention policy.
  8. Disable API termination protection only for the exact instance under approved teardown, then delete the stack.
  9. Confirm no orphan ENI, endpoint, role/profile, SG, route, volume, or unexpected retained log remains.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
AWS VPC private subnet routingofficialAWS Systems Manager VPC endpointsofficialAWS Session Manager getting startedofficialAWS Session Manager loggingofficialEC2 instance metadata options and IMDSv2officialEC2 EBS encryptionofficialAWS security group rulesofficialAWS IAM role for EC2officialAWS CloudFormation change setsofficialAWS CloudFormation rollbackofficialAWS CloudFormation drift detectionofficialAWS CloudTrail event historyofficialAmazon Linux 2023 EC2 launchofficial