Provision a hardened AWS EC2 web server in a private VPC
Deploy a private Amazon Linux 2023 web host with CloudFormation change sets, no public IP or SSH, SSM endpoints and Session Manager, explicit network paths, least-privilege role, IMDSv2, encrypted EBS, audit, drift, and rollback.
Create an evidence-backed EC2 host that can be administered and monitored without public management exposure or broad cloud/network authority.
- Amazon EC2 current regional API
- Amazon Linux 2023 reviewed AMI
- AWS CLI v2
- CloudFormation 2010-09-09 template format
- Governed AWS account and region Federated short-lived operator role, CloudTrail/Config/SCP guardrails, approved region/AZs, quotas, account alias, and cost/change owners.
aws sts get-caller-identity --query Arn --output text - Approved private network allocation Non-overlapping VPC/subnet CIDRs, internal DNS/resolver plan, client security group, regional S3 prefix list, and no requirement for public/NAT routing.
- Approved image and private management Reviewed AL2023/golden AMI, SSM Agent, endpoint service availability, Session Manager logging/preferences, and emergency serial-console procedure.
- Protected infrastructure repository Reviewed CloudFormation template/parameters, policy validation, secret scanning, exact change set approval, rollback owner, and no durable host data.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A CloudFormation-managed AWS VPC containing private application subnets without automatic public addresses or a default internet route, plus private AWS service endpoints required for Systems Manager administration.
- A hardened Amazon Linux 2023 EC2 web host with no inbound SSH, Session Manager access, IMDSv2 required, encrypted gp3 root storage, termination protection, a non-default application port, explicit security-group ingress and egress, and a narrowly scoped instance role.
- A preview-first deployment workflow using AWS CLI identity checks, template validation, CloudFormation change sets, resource drift and rollback evidence, Session Manager verification, CloudTrail, and service-level health.
- The instance has no public IPv4 address, no internet gateway/default route, no port 22 rule, no unrestricted ingress, and no unrestricted outbound internet path.
- Systems Manager reports the instance Online through VPC endpoints; operators start attributable sessions without distributing SSH keys or exposing an SSH daemon. CloudTrail records session lifecycle, while compatible interactive shell sessions can additionally stream content to the configured log group.
- Metadata requests without a token fail, token-authenticated metadata works only at hop limit one, every EBS volume is encrypted, and the web service answers only through the approved client security group and port.
- CloudFormation change sets expose every create/update/delete/replace before execution, stack rollback is enabled, CloudTrail records control-plane activity, and teardown requires explicit protection removal and evidence retention.
Architecture
How the parts fit together
CloudFormation declares a dedicated VPC and two private subnets across availability zones. Their route tables contain only local and S3 gateway routes; there is no internet gateway, NAT gateway, or automatic public IPv4 assignment. Interface VPC endpoints for Systems Manager, Session Manager messages, and CloudWatch Logs place private service interfaces in the VPC with private DNS; an EC2 Messages endpoint is added only when the selected region and SSM Agent generation demonstrably require it. The instance security group accepts the web port only from an approved client or load-balancer security group. Egress permits TLS to endpoint interfaces and TLS to the regional S3 managed prefix list for signed Amazon Linux package content; it does not permit 0.0.0.0/0. AmazonProvidedDNS traffic cannot be filtered by security groups, so domain filtering is a separate Route 53 Resolver DNS Firewall control rather than a misleading port 53 rule. The EC2 instance runs Amazon Linux 2023 from an approved SSM public parameter, requires IMDSv2, encrypts its gp3 root volume, uses no key pair, and has API termination protection. A custom instance role enumerates only current SSM channel and logging actions; unavoidable Resource-star operations are documented because those APIs do not support resource-level ARNs, never used as an action/principal wildcard shortcut.
- Authenticate the operator with a short-lived role, confirm account alias and region without recording the numeric account ID, and capture quotas and existing CIDR ownership.
- Review the CloudFormation template, parameters, IAM actions, endpoint set, image parameter, user data, security groups, storage, metadata, tags, deletion behavior, and outputs.
- Validate the template, create a change set, wait for completion, and inspect every resource replacement, IAM capability, and policy before execution.
- Execute inside an approved window and wait for stack completion; on failure preserve stack events and let CloudFormation roll back rather than patching resources manually.
- Prove private routing, no public IP/key pair/SSH, endpoint health, SSM Online status, token-required metadata, volume encryption, web health, logs, and CloudTrail evidence.
- Apply updates through new change sets and rehearse stack rollback. Teardown removes termination protection only after retention, dependency, and data checks.
Assumptions
- The deployment uses one approved AWS account and region, two available zones, non-overlapping RFC1918 CIDRs, and organization guardrails such as SCPs, CloudTrail, Config, and central log retention.
- AWS CLI v2 runs with a short-lived federated role. No access key, secret key, session token, numeric account ID, or private internal hostname is embedded in the template or tutorial evidence.
- Amazon Linux 2023 is approved and the selected public SSM parameter is resolved and recorded at change time. Production may replace it with a private golden-image parameter after image scanning and provenance.
- The web host is reached from an existing approved client/load-balancer security group inside connected private networking. Public exposure requires a separate edge/load-balancer/TLS/WAF design.
- The region supports the required Systems Manager interface endpoints. SSM Agent is present and running in the approved image, and private DNS/resolver behavior is functional.
- The stack contains no durable application data. Persistent data belongs on a separately backed-up managed service or volume with retention/deletion policy and recovery tests.
Key concepts
- Private subnet
- A subnet whose route table has no direct route to an internet gateway. A private address alone is not proof of private routing.
- Session Manager
- A Systems Manager capability that opens authenticated sessions through the SSM Agent and service control channels without inbound SSH or distributed host keys.
- IMDSv2
- The token-based EC2 Instance Metadata Service flow. Requiring tokens reduces exposure to unauthenticated metadata requests; hop limit one reduces proxy/container forwarding.
- VPC endpoint
- Private connectivity to supported AWS services. Interface endpoints use ENIs and security groups; gateway endpoints add service routes for S3 or DynamoDB.
- CloudFormation change set
- A preview of stack actions including replacement and policy changes. It does not prove workload correctness or absence of runtime side effects.
- No wildcard shortcut
- Actions, principals, accounts, and resource-capable APIs are enumerated. Resource '*' appears only for specifically enumerated APIs that expose no resource-level authorization and is documented and constrained by other layers.
Before you copy
Values used in this guide
{{awsRegion}}Approved AWS region selected by the operator profile.
Example: eu-central-1{{stackName}}Stable CloudFormation stack name for the private web host.
Example: production-private-web{{vpcCidr}}Approved non-overlapping VPC CIDR.
Example: 10.72.0.0/20{{privateSubnetACidr}}First private subnet CIDR.
Example: 10.72.0.0/24{{privateSubnetBCidr}}Second private subnet CIDR used by endpoints/resilience.
Example: 10.72.1.0/24{{clientSecurityGroupId}}Existing security group representing the approved internal load balancer or client tier.
Example: sg-REDACTED_APPROVED_CLIENT{{s3PrefixListId}}Regional AWS-managed S3 prefix-list ID resolved before deployment.
Example: pl-REDACTED_REGIONAL_S3{{instanceType}}Approved architecture/capacity class supported by the selected image.
Example: t3.small{{imageParameter}}SSM parameter resolving to the reviewed Amazon Linux 2023 image.
Example: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64{{releaseVersion}}Non-secret change identifier used to name reviewed CloudFormation change sets and parameter snapshots.
Example: 2026.07.28-1Security and production boundaries
- Do not add port 22, a key pair, public IP, internet gateway, NAT route, or 0.0.0.0/0 egress as a troubleshooting shortcut. Diagnose endpoint, DNS, IAM, and agent state through console/SSM evidence.
- The instance role contains only enumerated SSM message and logging actions. Where AWS APIs require Resource '*', document the limitation and constrain account, region, endpoint, network, SCP, and monitoring layers.
- IMDSv2 does not make metadata credentials harmless. Limit the role, prevent untrusted workloads from reaching metadata, use hop limit one, and patch SSRF vulnerabilities.
- User data is visible to identities with EC2 read permissions and can appear in logs. It must contain no secret, token, certificate private key, account ID, or application credential.
- CloudFormation parameters and outputs are not secret stores. Use Secrets Manager or Parameter Store with workload identity and avoid exposing dynamic references in logs.
- Session Manager removes inbound SSH but is privileged remote execution. Require SSO/MFA, least-privilege session permissions, logging, session duration, approval, and alerts.
Stop before continuing if
- Stop if the caller identity, account alias, region, CIDR ownership, endpoint support, or organization guardrails cannot be verified.
- Stop if the change set creates a public IP, internet route, port 22, unrestricted ingress/egress, Action/Principal wildcard, unencrypted volume, optional IMDS tokens, or a literal secret.
- Stop if the selected image or package origin is unverified, the instance role is broader than SSM/logging, or user data contains sensitive material.
- Stop if any resource replacement affects retained data or a shared network boundary without a tested migration.
- Stop if SSM does not become Online, metadata v1 still works, EBS is unencrypted, CloudTrail is unavailable, or web health fails.
verification
Verify short-lived AWS identity, alias, region, and quotas
Use a federated CLI profile, confirm role ARN and account alias without copying the numeric account ID into records, set the approved region explicitly, and inspect relevant VPC/endpoint/instance quotas before composing a change.
Why this step matters
The most damaging cloud mistake is a correct command in the wrong account or region. A role ARN and human-readable alias create two identity signals while evidence redacts the numeric account. Short-lived federation limits credential lifetime. Quotas are part of design because endpoint creation can fail after partial stack work. The operator role should create only reviewed network/EC2/IAM resources and should not itself be the instance role.
What to understand
Use AWS_PROFILE and AWS_REGION explicitly in automation rather than relying on a stale default.
Confirm SCP, permission boundary, CloudTrail, Config, and change-window controls.
Record CIDR allocation from the authoritative IPAM source.
System changes
- Queries identity, alias, configuration, and quotas only.
Syntax explained
--query Arn- Returns the active principal without dumping temporary credentials.
service-quotas get-service-quota- Reads a specific regional/service limit before deployment.
aws sts get-caller-identity --profile production-operator --query Arn --output text && aws iam list-account-aliases --profile production-operator --query 'AccountAliases[0]' --output text && aws configure get region --profile production-operator && aws service-quotas get-service-quota --service-code vpc --quota-code L-29B6F2EB --profile production-operatorarn:aws:sts::REDACTED:assumed-role/NetworkChangeOperator/session-redacted
production-platform
eu-central-1
{
"Quota": {
"QuotaName": "Interface VPC endpoints per VPC",
"Value": 50.0
}
}Checkpoint: Cloud scope is unambiguous
aws sts get-caller-identity --query Arn --output text && aws ec2 describe-availability-zones --filters Name=state,Values=available --query 'AvailabilityZones[].ZoneName' --output textContinue whenApproved federated role, production alias, region, two available zones, sufficient quotas, and non-overlapping IPAM allocation.
Stop whenIdentity, alias, region, credential lifetime, guardrails, availability zones, quota, or CIDR differs from approval.
If this step fails
Account alias is empty or unexpected.
Likely causeWrong account or alias not configured.
Confirm through organization account inventory.
ResolutionStop and select the correct federated account/profile.
Security notes
- Redact numeric account IDs and session names in externally shared evidence.
Alternatives
- Use AWS IAM Identity Center named profiles with enforced MFA.
Stop conditions
- Stop on any scope ambiguity.
verification
Resolve the approved image and regional endpoint dependencies
Resolve the Amazon Linux 2023 SSM parameter to one AMI, inspect its owner and deprecation state, enumerate Systems Manager endpoint services, and resolve the regional S3 managed prefix list. Record identifiers in protected change evidence, not source code.
Why this step matters
A friendly SSM parameter moves as new images publish. Resolving it at preview time and recording the concrete AMI makes the change auditable. The image owner must be Amazon and state available; organizations should add their own scan/sign pipeline. Endpoint availability differs by region and generation. SSM uses ssm and ssmmessages, with ec2messages still relevant in supported environments. S3 prefix lists support explicit security-group egress for package repositories without arbitrary internet access.
What to understand
Treat AMI ID, owner, architecture, boot mode, deprecation, and root device as one reviewed set.
Use a private golden-image parameter for mature production.
Do not hard-code endpoint service names from another region.
System changes
- Queries public/service metadata only.
Syntax explained
SSM public AMI parameter- Returns the current AL2023 image ID; record the resolved value.
managed prefix list- AWS-maintained regional destination set for S3 routing/rules.
Values stay on this page and are never sent or saved.
AMI_ID=$(aws ssm get-parameter --name '{{imageParameter}}' --query 'Parameter.Value' --output text) && aws ec2 describe-images --image-ids "$AMI_ID" --query 'Images[0].[ImageId,OwnerId,Name,State,DeprecationTime]' --output table && aws ec2 describe-vpc-endpoint-services --query "ServiceNames[?contains(@, '.ssm') || contains(@, '.ssmmessages') || contains(@, '.ec2messages') || contains(@, '.logs')]" --output text && aws ec2 describe-managed-prefix-lists --filters Name=prefix-list-name,Values=com.amazonaws.{{awsRegion}}.s3 --query 'PrefixLists[0].[PrefixListId,PrefixListName]' --output text-------------------------------------------------------------- | DescribeImages | +----------------------+----------+---------------------------+ | ami-REDACTED | amazon | al2023-ami-2026.07... | | available | None | | +----------------------+----------+---------------------------+ com.amazonaws.eu-central-1.ssm com.amazonaws.eu-central-1.ssmmessages com.amazonaws.eu-central-1.ec2messages com.amazonaws.eu-central-1.logs pl-REDACTED com.amazonaws.eu-central-1.s3
Checkpoint: Image and endpoint dependencies are supported and pinned for review
aws ec2 describe-images --image-ids "$AMI_ID" --query 'Images[0].[State,Public,RootDeviceType,Architecture]' --output textContinue whenImage is available, EBS-backed, expected architecture, owned by Amazon/approved pipeline, and every endpoint service exists.
Stop whenImage owner/state/architecture is wrong, deprecated unexpectedly, or required endpoint service/prefix list is absent.
If this step fails
Public image parameter resolves to a different AMI during review.
Likely causeA new AL2023 image was published.
Compare parameter LastModifiedDate and both image records.
ResolutionRestart image review and change-set generation with one concrete approved AMI.
Security notes
- AMI IDs are not secrets but can reveal patch cadence and architecture.
Alternatives
- Use a signed private golden-image SSM parameter.
Stop conditions
- Stop if image provenance is not accepted.
config
Declare private subnets with no internet route
Create the VPC, two private subnets, DNS support, and dedicated route tables. Do not attach an internet gateway or NAT gateway and do not enable automatic public addressing. Add the S3 gateway endpoint to both route tables.
Why this step matters
Subnet names do not create privacy; routing and interface addressing do. MapPublicIpOnLaunch false prevents automatic public addresses, while absence of internet/NAT routes removes generic outbound internet. Separate route tables make the boundary inspectable. The S3 gateway endpoint adds regional service routes for package content but not arbitrary destinations. DNS support is required for interface endpoint private DNS and normal service discovery.
What to understand
Validate CIDRs against IPAM, peering, transit gateway, on-premises, and service networks.
Network ACLs remain default stateful-compatible; add custom NACLs only with tested return paths.
Tag ownership, environment, data classification, and cost center in production.
System changes
- Template validation only; later stack execution creates the network.
Syntax explained
MapPublicIpOnLaunch: false- Prevents automatic public IPv4 assignment in the subnet.
Gateway endpoint RouteTableIds- Adds S3 service routes only to the selected private route tables.
infrastructure/private-web.ymlAWSTemplateFormatVersion: "2010-09-09"
Description: Private SSM-managed web host
Parameters:
VpcCidr: {Type: String}
PrivateSubnetACidr: {Type: String}
PrivateSubnetBCidr: {Type: String}
ClientSecurityGroupId: {Type: AWS::EC2::SecurityGroup::Id}
S3PrefixListId: {Type: String}
InstanceType: {Type: String, Default: t3.small}
ImageId: {Type: AWS::EC2::Image::Id}
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCidr
EnableDnsSupport: true
EnableDnsHostnames: true
Tags: [{Key: Name, Value: private-web}]
PrivateSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
CidrBlock: !Ref PrivateSubnetACidr
AvailabilityZone: !Select [0, !GetAZs ""]
MapPublicIpOnLaunch: false
PrivateSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
CidrBlock: !Ref PrivateSubnetBCidr
AvailabilityZone: !Select [1, !GetAZs ""]
MapPublicIpOnLaunch: false
RouteTableA:
Type: AWS::EC2::RouteTable
Properties: {VpcId: !Ref Vpc}
RouteTableB:
Type: AWS::EC2::RouteTable
Properties: {VpcId: !Ref Vpc}
AssociationA:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties: {SubnetId: !Ref PrivateSubnetA, RouteTableId: !Ref RouteTableA}
AssociationB:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties: {SubnetId: !Ref PrivateSubnetB, RouteTableId: !Ref RouteTableB}
S3Endpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
VpcId: !Ref Vpc
ServiceName: !Sub com.amazonaws.${AWS::Region}.s3
VpcEndpointType: Gateway
RouteTableIds: [!Ref RouteTableA, !Ref RouteTableB]aws cloudformation validate-template --template-body file://private-web.yml --query 'Parameters[].ParameterKey' --output textVpcCidr PrivateSubnetACidr PrivateSubnetBCidr ClientSecurityGroupId S3PrefixListId InstanceType ImageId
Checkpoint: Network template has no internet path
grep -nE 'InternetGateway|NatGateway|0\.0\.0\.0/0|MapPublicIpOnLaunch: true' infrastructure/private-web.yml || trueContinue whenNo output and both private subnets have dedicated route tables plus S3 endpoint routes.
Stop whenAny public addressing, internet/NAT route, shared route table, or overlapping CIDR appears.
If this step fails
CloudFormation reports CIDR overlap.
Likely causeVPC or subnet allocation conflicts.
Compare IPAM and existing VPC/subnet CIDRs.
ResolutionAllocate new approved non-overlapping ranges; never squeeze or reuse ranges ad hoc.
Security notes
- Private routes reduce exposure but do not replace SG, IAM, patching, or application auth.
Alternatives
- Attach to an existing governed private VPC after ownership review.
Stop conditions
- Stop if private routing cannot be proven.
config
Create private service endpoints and explicit network rules
Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Create interface endpoints in both private subnets with private DNS. The endpoint group accepts TLS only from the instance group. The instance accepts the web port only from the approved client group and sends TLS only to endpoint interfaces and the regional S3 prefix list. AmazonProvidedDNS remains reachable independently of security-group rules; use Route 53 Resolver DNS Firewall when DNS-domain filtering is required.
Why this step matters
Security groups are stateful, so return traffic does not need mirror rules. Referencing groups expresses workload relationships more safely than broad CIDRs. Removing default egress is essential because AWS-created groups otherwise allow all outbound traffic. Endpoint TLS goes only to endpoint ENIs, and S3 TLS goes only to the AWS-managed regional prefix list. AmazonProvidedDNS traffic cannot be blocked or filtered by security groups, so a VPC-CIDR port 53 rule would create false assurance; use Route 53 Resolver DNS Firewall for domain filtering. Current SSM Agent versions prefer ssm and ssmmessages. Add an ec2messages endpoint and the matching minimal IAM actions only when AWS regional endpoint guidance and a tested agent version show that the fallback channel is required.
What to understand
AmazonProvidedDNS bypasses security-group and network-ACL filtering. Verify Route 53 Resolver DNS Firewall policy associations or custom resolver paths separately when DNS control is a requirement.
Use endpoint policies to narrow services where supported; SGs govern network, not API authorization.
The client SG must belong to the same connected security boundary and be separately controlled.
All fragments in this tutorial are parts of one infrastructure/private-web.yml template; merge them under the existing Resources map before validation and change-set creation.
System changes
- Template review only; stack later creates ENIs and security rules.
Syntax explained
SourceSecurityGroupId- Allows only network interfaces associated with the approved source group.
DestinationPrefixListId- Restricts outbound TLS to AWS-managed regional S3 addresses.
PrivateDnsEnabled- Resolves standard AWS service names to private endpoint addresses in the VPC.
infrastructure/private-web.ymlEndpointSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: TLS from private web instances to AWS endpoints
VpcId: !Ref Vpc
SecurityGroupEgress: []
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Private web host explicit paths
VpcId: !Ref Vpc
SecurityGroupEgress: []
EndpointTlsIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref EndpointSecurityGroup
IpProtocol: tcp
FromPort: 443
ToPort: 443
SourceSecurityGroupId: !Ref InstanceSecurityGroup
PrivateWebIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref InstanceSecurityGroup
IpProtocol: tcp
FromPort: 8080
ToPort: 8080
SourceSecurityGroupId: !Ref ClientSecurityGroupId
EndpointTlsEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !Ref InstanceSecurityGroup
IpProtocol: tcp
FromPort: 443
ToPort: 443
DestinationSecurityGroupId: !Ref EndpointSecurityGroup
S3TlsEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !Ref InstanceSecurityGroup
IpProtocol: tcp
FromPort: 443
ToPort: 443
DestinationPrefixListId: !Ref S3PrefixListId
SsmEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
VpcId: !Ref Vpc
ServiceName: !Sub com.amazonaws.${AWS::Region}.ssm
VpcEndpointType: Interface
PrivateDnsEnabled: true
SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
SecurityGroupIds: [!Ref EndpointSecurityGroup]
SsmMessagesEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
VpcId: !Ref Vpc
ServiceName: !Sub com.amazonaws.${AWS::Region}.ssmmessages
VpcEndpointType: Interface
PrivateDnsEnabled: true
SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
SecurityGroupIds: [!Ref EndpointSecurityGroup]
LogsEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
VpcId: !Ref Vpc
ServiceName: !Sub com.amazonaws.${AWS::Region}.logs
VpcEndpointType: Interface
PrivateDnsEnabled: true
SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
SecurityGroupIds: [!Ref EndpointSecurityGroup]aws cloudformation validate-template --template-body file://private-web.yml >/dev/null && grep -nE 'SecurityGroup(Ingress|Egress)|VPCEndpoint' infrastructure/private-web.yml88: EndpointSecurityGroup: 101: InstanceSecurityGroup: 135: SsmEndpoint: 145: SsmMessagesEndpoint: 155: LogsEndpoint:
Checkpoint: Only the documented network relationships exist
grep -nE '0\.0\.0\.0/0|::/0|FromPort: 22|IpProtocol: -1' infrastructure/private-web*.yml || trueContinue whenNo output; web ingress is client-SG-to-instance:8080 and security-group egress is only endpoint TLS and S3 TLS. AmazonProvidedDNS behavior and any Resolver DNS Firewall policy are verified separately.
Stop whenAny unrestricted rule, SSH, public CIDR, default egress, endpoint without private DNS, or unowned client group appears.
If this step fails
Endpoint DNS resolves publicly from the instance.
Likely causePrivate DNS disabled, VPC DNS disabled, or custom resolver bypasses endpoint records.
Resolve service names through approved diagnostics and inspect endpoint DNS entries.
ResolutionCorrect VPC/resolver/private-DNS design before starting SSM sessions.
Security notes
- Endpoint security groups do not replace endpoint policies and IAM.
Alternatives
- Use centrally shared endpoints through a governed network architecture.
Stop conditions
- Stop if management needs unrestricted internet egress.
config
Create an enumerated SSM and logging instance role
Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Declare an EC2 trust relationship and list only the current SSM channel, inventory, and log actions required. Some SSM and CloudWatch Logs discovery APIs do not support resource-level ARNs; each Resource '*' is an explicit service limitation paired with enumerated actions, private endpoints, region/account guardrails, and monitoring—not a wildcard shortcut.
Why this step matters
An EC2 role supplies temporary credentials; it should not contain a human login permission or application secrets. The trust policy names only the EC2 service principal. Actions are enumerated. Resource '*' is retained only for listed SSM/channel actions and logs:DescribeLogGroups, whose authorization models have no usable resource ARN; both exceptions must survive Access Analyzer review. Log stream creation and writes are scoped to one log group ARN. The default example omits legacy ec2messages permissions because current SSM Agent versions prefer ssmmessages; add only the exact fallback actions after verifying that the selected region and agent require them. Organization SCPs, endpoint policy, region controls, instance profile association, and CloudTrail further limit and observe use.
What to understand
Do not attach AdministratorAccess or a broad convenience policy.
Separate application data permissions into another narrowly reviewed statement/role only when required.
Validate both identity and trust policies; Access Analyzer findings require disposition, not blind suppression.
System changes
- Validates a policy document only; stack later creates role/profile.
Syntax explained
Principal ec2.amazonaws.com- Allows the EC2 service to deliver role credentials to associated instances.
enumerated Action list- Prevents service-wide or account-wide action wildcards.
LogGroup.Arn:*- Scopes log streams/events beneath one controlled group.
infrastructure/private-web.ymlSessionLogGroup:
Type: AWS::Logs::LogGroup
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
LogGroupName: !Sub /aws/ssm/${AWS::StackName}
RetentionInDays: 90
InstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal: {Service: ec2.amazonaws.com}
Action: sts:AssumeRole
Policies:
- PolicyName: SsmMessagingAndInventory
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: SsmControlChannelsNoResourceArn
Effect: Allow
Action:
- ssmmessages:CreateControlChannel
- ssmmessages:CreateDataChannel
- ssmmessages:OpenControlChannel
- ssmmessages:OpenDataChannel
- ssm:UpdateInstanceInformation
Resource: "*"
- Sid: DiscoverConfiguredSessionLogGroup
Effect: Allow
Action: logs:DescribeLogGroups
Resource: "*"
- Sid: SessionLogsOnly
Effect: Allow
Action:
- logs:CreateLogStream
- logs:DescribeLogStreams
- logs:PutLogEvents
Resource: !Sub ${SessionLogGroup.Arn}:*
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles: [!Ref InstanceRole]aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://instance-policy.json --query 'findings[].[findingType,issueCode]' --output table--------------------------------- | ValidatePolicy | +--------------+----------------+ | findingType | issueCode | +--------------+----------------+ | SUGGESTION | CREATE_SLR_WITH_NOT_RESOURCE | +--------------+----------------+
Checkpoint: Role has no wildcard action/principal and every resource exception is justified
aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://instance-policy.jsonContinue whenNo ERROR/SECURITY_WARNING remains; SSM channel and logs:DescribeLogGroups Resource-star limitations are documented; log stream writes are scoped to the exact log group.
Stop whenAction/Principal wildcard, broad managed policy, unrelated service, write-capable application permission, or unexplained analyzer finding appears.
If this step fails
Analyzer warns about Resource '*'.
Likely causeThe enumerated SSM API does not support resource-level permissions or policy is broader than needed.
Check AWS Service Authorization reference for each action.
ResolutionKeep only unavoidable enumerated actions and add conditions/guardrails where supported; remove every resource-capable wildcard.
Security notes
- Metadata access exposes role credentials; least privilege remains mandatory even with IMDSv2.
Alternatives
- Use the current AWS-managed SSM core policy only after reviewing its exact version and broader statements; custom policy is easier to audit here.
Stop conditions
- Stop if SSM cannot operate without unrelated authority.
config
Require IMDSv2, encrypted gp3, no key pair, and protected lifecycle
Append this fragment beneath the same reviewed Resources map in infrastructure/private-web.yml. Launch one private Amazon Linux 2023 instance without a public IP or key pair. Require metadata tokens, hop limit one, disabled metadata tags, encrypted gp3 storage, detailed monitoring, termination protection, and minimal non-secret bootstrap.
Why this step matters
HttpTokens required rejects IMDSv1; hop limit one reduces forwarding beyond the instance network boundary. Instance metadata tags are disabled to avoid exposing tag content locally. Encrypted gp3 protects the root volume at rest; account default encryption is helpful but the template states intent. No KeyName means no SSH key is installed, and the SG has no SSH ingress. Termination protection prevents casual API deletion but must be deliberately removed for replacement/teardown. User data is intentionally secret-free and fails on error.
What to understand
Prefer a golden image so bootstrap does not depend on package repositories.
Use TLS at an internal load balancer or configure a private certificate through workload identity; never embed a key in user data.
Size storage and instance class through measured workload requirements.
System changes
- Template review only; stack later launches and bootstraps the instance.
Syntax explained
HttpTokens: required- Requires a session token for all metadata requests.
HttpPutResponseHopLimit: 1- Restricts metadata token responses to one network hop.
DisableApiTermination: true- Blocks termination until explicitly disabled.
infrastructure/private-web.ymlWebInstance:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref ImageId
InstanceType: !Ref InstanceType
IamInstanceProfile: {Name: !Ref InstanceProfile}
SubnetId: !Ref PrivateSubnetA
SecurityGroupIds: [!Ref InstanceSecurityGroup]
DisableApiTermination: true
Monitoring: true
MetadataOptions:
HttpEndpoint: enabled
HttpTokens: required
HttpPutResponseHopLimit: 1
InstanceMetadataTags: disabled
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
Encrypted: true
VolumeType: gp3
VolumeSize: 16
DeleteOnTermination: true
Tags:
- {Key: Name, Value: private-web}
- {Key: ManagedBy, Value: cloudformation}
UserData:
Fn::Base64: |
#!/bin/bash
set -euo pipefail
dnf -y install nginx
cat >/etc/nginx/conf.d/private-web.conf <<'CONF'
server {
listen 8080;
server_name _;
server_tokens off;
location = /healthz { return 200 "healthy
"; }
location / { return 404; }
}
CONF
nginx -t
systemctl enable --now nginxaws cloudformation validate-template --template-body file://private-web.yml >/dev/null && grep -nE 'MetadataOptions|Encrypted|AssociatePublicIpAddress|DisableApiTermination|KeyName' infrastructure/private-web.yml214: DisableApiTermination: true 221: AssociatePublicIpAddress: false 229: MetadataOptions: 235: Encrypted: true
Checkpoint: Launch declaration has every hardening invariant
grep -nE 'HttpTokens: required|HttpPutResponseHopLimit: 1|Encrypted: true|AssociatePublicIpAddress: false|DisableApiTermination: true' infrastructure/private-web*.ymlContinue whenAll invariants are present, no KeyName/public IP/secret exists, and image plus bootstrap are reviewed.
Stop whenMetadata tokens optional, storage plaintext, public addressing, SSH key, secret user data, mutable image, or termination protection absent.
If this step fails
User data finishes partially.
Likely causePackage/DNS/S3 endpoint failure or nginx validation error.
Read cloud-init output and systemd status through SSM/serial console.
ResolutionCorrect template/image and replace through a change set; do not manually complete hidden drift.
Security notes
- EC2 read-user-data permission exposes bootstrap; keep it non-sensitive.
Alternatives
- Use a scanned golden AMI with no bootstrap package installation.
Stop conditions
- Stop if bootstrap cannot complete without broad egress or secrets.
command
Create a CloudFormation change set and inspect every action
Validate the final template, create a named change set with explicit parameters and IAM capability, wait for creation, then inspect action, replacement, scope, details, and policy changes. Do not execute from a pipeline that hides the change set.
Why this step matters
CloudFormation validation checks template structure but not architectural safety. A change set reveals concrete resource actions and whether an update replaces a resource. CAPABILITY_NAMED_IAM is explicit acknowledgement that the stack creates named IAM resources; it is not authorization to skip policy review. Parameter files can expose topology and must not contain secrets. CREATE versus UPDATE type must match stack state.
What to understand
Review all parameters, tags, resolved AMI, endpoint services, SG rules, role policies, user data, outputs, deletion/update policies, and replacements.
Run Access Analyzer on extracted policies and organization policy-as-code before execution.
Delete stale unexecuted change sets to prevent accidental later execution.
System changes
- Creates a CloudFormation change-set preview but no stack resources.
Syntax explained
CAPABILITY_NAMED_IAM- Acknowledges creation/modification of explicitly named IAM resources.
Replacement- Shows whether update requires a new physical resource.
Values stay on this page and are never sent or saved.
aws cloudformation validate-template --template-body file://infrastructure/private-web.yml && aws cloudformation create-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --change-set-type CREATE --template-body file://infrastructure/private-web.yml --parameters file://infrastructure/parameters.json --capabilities CAPABILITY_NAMED_IAM --query Id --output text && aws cloudformation wait change-set-create-complete --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} && aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --query 'Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement,Scope]' --output tableTemplate validation: parameters accepted arn:aws:cloudformation:eu-central-1:REDACTED:changeSet/reviewed-2026.07.28-1/REDACTED -------------------------------------------------------------------------------- | DescribeChangeSet | +--------+------------------------+------------------------------+-------+-------+ | Add | Vpc | AWS::EC2::VPC | False | None | | Add | SsmEndpoint | AWS::EC2::VPCEndpoint | False | None | | Add | InstanceRole | AWS::IAM::Role | False | None | | Add | WebInstance | AWS::EC2::Instance | False | None |
Checkpoint: Change set contains only accepted resources and no hidden replacement
aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} --output json > /secure-staging/change-set.json && sha256sum /secure-staging/change-set.jsonContinue whenEvery action, IAM statement, network rule, image, storage property, and replacement is accepted and snapshot digest recorded.
Stop whenUnexpected delete/replace, wildcard, public route/address, plaintext storage, broad role, secret parameter, or unreviewed resource appears.
If this step fails
Change set status is FAILED with no changes.
Likely causeTemplate equals live stack or wrong CREATE/UPDATE type.
Read StatusReason and stack existence.
ResolutionTreat no changes as evidence; do not force an unrelated mutation.
Security notes
- Change-set JSON can reveal internal network and IAM design; protect it.
Alternatives
- Use CloudFormation StackSets with delegated admin after the same per-account preview controls.
Stop conditions
- Stop until a human reviewer approves exact physical actions.
command
Execute the approved change set and preserve stack events
Execute only the reviewed change-set ID, wait for stack completion, and export stack events and outputs with account identifiers redacted. Let CloudFormation roll back on failure; do not disable rollback for production.
Why this step matters
The change-set name and stored digest bind execution to review. CloudFormation orders dependencies and records each resource event. A stack CREATE_COMPLETE means infrastructure APIs succeeded, not that SSM, metadata, web health, logging, and security invariants work. Rollback on failure should remain enabled so partially created resources are removed where possible, but event evidence must be captured first.
What to understand
Monitor the first failure event, not only final stack status.
Avoid manual fixes because they create drift and may break rollback.
Redact account IDs and sensitive internal outputs from external evidence.
System changes
- Creates VPC, subnets, endpoints, SGs, IAM role/profile, logs, and EC2 instance.
Syntax explained
execute-change-set- Applies the previously generated concrete change plan.
wait stack-create-complete- Returns only after terminal create success/failure.
Values stay on this page and are never sent or saved.
aws cloudformation execute-change-set --stack-name {{stackName}} --change-set-name reviewed-{{releaseVersion}} && aws cloudformation wait stack-create-complete --stack-name {{stackName}} && aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].[StackStatus,Outputs]' --output json && aws cloudformation describe-stack-events --stack-name {{stackName}} --max-items 30 --output table[
"CREATE_COMPLETE",
[
{"OutputKey": "InstanceId", "OutputValue": "i-REDACTED"},
{"OutputKey": "VpcId", "OutputValue": "vpc-REDACTED"}
]
]
CREATE_COMPLETE AWS::CloudFormation::Stack production-private-web
CREATE_COMPLETE AWS::EC2::Instance WebInstance
CREATE_COMPLETE AWS::EC2::VPCEndpoint SsmEndpointCheckpoint: Stack API operation completes without manual drift
aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output textContinue whenCREATE_COMPLETE and no manual resource edit or unresolved warning event.
Stop whenRollback begins, resource fails, endpoint/role differs, or anyone proposes console patching outside the template.
If this step fails
Stack rolls back after instance creation.
Likely causeEndpoint, IAM, subnet, bootstrap, or wait-condition dependency failed.
Preserve ordered stack events and first failure reason.
ResolutionLet rollback complete, correct source template, create a new change set.
Security notes
- CloudFormation events can include property identifiers; protect exports.
Alternatives
- Execute through a protected CI role after human change-set approval.
Stop conditions
- Stop post-deploy acceptance until stack is stable.
verification
Prove no public IP, internet route, SSH, or unrestricted rule exists
Describe the instance interfaces, subnet public-address setting, route tables, and all SG rules by rule ID. Verify no internet gateway/NAT route, public address, key pair, port 22, or all-address/all-protocol rule exists.
Why this step matters
Template intent can be changed by defaults, drift, or another tool. Runtime inspection proves actual interfaces, routes, and SG rules. KeyName None confirms no EC2 key pair was injected; it does not prove sshd package absence, so port/process checks follow through SSM. Rule IDs support exact revocation and CloudTrail correlation.
What to understand
Inspect every ENI, not only primary interface.
Check subnet MapPublicIpOnLaunch independently.
Run CloudFormation drift after runtime inspection.
System changes
- Reads EC2/VPC state only.
Syntax explained
PublicIpAddress None- No public IPv4 is associated with the instance.
route target vpce- S3 service traffic uses the gateway endpoint rather than internet.
Values stay on this page and are never sent or saved.
INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name {{stackName}} --query "Stacks[0].Outputs[?OutputKey=='InstanceId'].OutputValue" --output text) && aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[PrivateIpAddress,PublicIpAddress,KeyName,MetadataOptions.HttpTokens]' --output table && aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].SubnetId' --output text) --query 'RouteTables[0].Routes' --output table && aws ec2 describe-security-group-rules --filters Name=group-id,Values=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupId' --output text) --output table------------------------------------------------ | DescribeInstances | +-------------+--------+-------+----------------+ | 10.72.0.34 | None | None | required | +-------------+--------+-------+----------------+ | Destination | Target | State | | 10.72.0.0/20| local | active | | pl-REDACTED | vpce-REDACTED | active | No rules matching port 22, 0.0.0.0/0, ::/0, or protocol -1
Checkpoint: Runtime network is private and explicit
aws cloudformation detect-stack-drift --stack-name {{stackName}} --query StackDriftDetectionId --output textContinue whenNo public address/key, no default internet route, no SSH/broad rules, and later drift result is IN_SYNC.
Stop whenAny exposure, unknown ENI/route/rule, or drift exists.
If this step fails
Unexpected SG rule exists outside template.
Likely causeManual change or another automation owner.
Correlate rule ID with CloudTrail and stack drift.
ResolutionTreat as incident, revoke under approval, and restore single ownership.
Security notes
- Network identifiers are sensitive operational metadata.
Alternatives
- Use AWS Config rules for continuous public-exposure detection.
Stop conditions
- Stop acceptance on any unreviewed reachability.
verification
Verify SSM Online status and an attributable no-SSH session
Wait for the instance to register as a managed node, inspect agent/platform data, start one approved Session Manager shell session, run non-mutating identity and service checks, then terminate it and correlate the CloudTrail event and compatible session-content log. CloudTrail records session lifecycle, but SSH and port-forwarding sessions through Session Manager do not support command-content logging.
Why this step matters
Online status proves the agent, role, DNS, endpoints, and service control channels are functioning. Session Manager authenticates the operator through IAM and avoids inbound SSH keys, but it still grants interactive host access and often sudo through ssm-user configuration. Session policy, logging, MFA, duration, approvals, and alerting must constrain it. Use the session only for read-only acceptance; durable configuration belongs in the image/template.
What to understand
Configure Session Manager preferences to CloudWatch Logs/S3 with encryption before routine use.
Restrict start-session to tagged instances and approved documents.
Alert on StartSession, ResumeSession, and TerminateSession.
Do not describe SSH or port-forwarding sessions as content-audited: Session Manager cannot log their commands or tunneled traffic. Use only session lifecycle events for those modes.
System changes
- Starts and terminates an attributable interactive shell session; commands shown are read-only. Compatible content logging is verified separately from CloudTrail lifecycle logging.
Syntax explained
describe-instance-information- Shows managed-node connectivity and agent/platform metadata.
start-session --target- Opens a privileged management channel; requires explicit IAM authorization.
aws ssm describe-instance-information --filters Key=InstanceIds,Values="$INSTANCE_ID" --query 'InstanceInformationList[0].[PingStatus,PlatformName,PlatformVersion,AgentVersion]' --output table && aws ssm start-session --target "$INSTANCE_ID"------------------------------------------------- | DescribeInstanceInformation | +--------+------------------+----------+---------+ | Online | Amazon Linux | 2023 | 3.3.x | +--------+------------------+----------+---------+ Starting session with SessionId: operator-REDACTED sh-5.2$ id uid=1000(ssm-user) gid=1000(ssm-user) groups=1000(ssm-user) sh-5.2$ systemctl is-active nginx active sh-5.2$ exit Exiting session.
Checkpoint: Management works without inbound SSH and produces audit evidence
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=StartSession --max-results 5 --query 'Events[0].[EventName,Username]' --output tableContinue whenNode Online, session identity approved, no port 22/key pair, session terminated, CloudTrail lifecycle event present, and content log present only for the compatible shell-session mode tested here.
Stop whenLogging missing, session identity/permissions broader than policy, node Offline, or anyone proposes SSH fallback.
If this step fails
StartSession denied while node is Online.
Likely causeOperator IAM lacks target/document/session permissions or conditions mismatch tags.
Inspect encoded authorization message and simulate operator policy.
ResolutionGrant only the exact session document/target conditions; do not broaden instance role.
Security notes
- Interactive sessions are privileged and may expose host data; minimize and audit.
Alternatives
- Use Run Command with an allowlisted read-only document for routine diagnostics.
Stop conditions
- Stop if session cannot be logged and attributed.
verification
Test IMDSv2 enforcement, EBS encryption, listeners, and bootstrap
Through the approved, attributable shell session, require an unauthenticated metadata request to fail, obtain a short-lived token for a permitted request, inspect nginx, listeners, package signature state, and cloud-init. From the control plane, verify every attached EBS volume is encrypted.
Why this step matters
Control-plane metadata options and an on-host negative probe together prove enforcement. A 401 without token is expected. Token lifetime should be short for diagnostics and token output never logged. Every attached volume, including later data volumes, must be checked. Nginx configuration validation and listener inspection prove the service uses the intended port, while cloud-init status and package verification establish bootstrap completion.
What to understand
Never print role credentials from metadata.
Check unexpected listeners and sshd state; no ingress rule is not permission to ignore host services.
Verify KMS key policy when a customer-managed key replaces aws/ebs.
System changes
- Performs metadata/storage/service reads only.
Syntax explained
HttpTokens required- Rejects metadata requests that omit a valid PUT token.
hop limit 1- Limits token response forwarding beyond the host network context.
aws ec2 describe-volumes --filters Name=attachment.instance-id,Values="$INSTANCE_ID" --query 'Volumes[].{Id:VolumeId,Encrypted:Encrypted,Type:VolumeType,KmsKeyId:KmsKeyId}' --output table && aws ec2 describe-instance-metadata-options --instance-ids "$INSTANCE_ID" --query 'InstanceMetadataOptions[0].[HttpTokens,HttpPutResponseHopLimit,InstanceMetadataTags,State]' --output table--------------------------------------------------------
| DescribeVolumes |
+--------------+-----------+------+--------------------+
| vol-REDACTED | True | gp3 | alias/aws/ebs ... |
+--------------+-----------+------+--------------------+
------------------------------------------------
| DescribeInstanceMetadataOptions |
+----------+--------+----------+----------------+
| required | 1 | disabled | applied |
+----------+--------+----------+----------------+
IMDSv1 probe: HTTP 401
nginx: configuration file /etc/nginx/nginx.conf test is successful
LISTEN 0 511 0.0.0.0:8080 users:(("nginx",pid=1284,fd=6))Checkpoint: Runtime matches metadata, disk, and service hardening
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[State.Name,MetadataOptions.HttpTokens,MetadataOptions.HttpPutResponseHopLimit]' --output textContinue whenInstance running, tokens required, hop limit one, every EBS encrypted, IMDSv1 401, expected listener only, bootstrap and nginx validation successful.
Stop whenMetadata v1 succeeds, credential output appears, any volume plaintext, unexpected listener/package, or cloud-init failed.
If this step fails
Volume encryption true but unexpected KMS key.
Likely causeAccount default key, launch template, or copied AMI snapshot uses another key.
Inspect volume, snapshot, default encryption, and key policy metadata.
ResolutionValidate key ownership/policy or replace through encrypted snapshot migration.
Security notes
- Never retrieve /latest/meta-data/iam/security-credentials role values for testing.
Alternatives
- Disable metadata endpoint entirely if the workload and agent do not require it.
Stop conditions
- Stop on any failed hardening invariant.
verification
Verify the web path from the approved client tier
Resolve the instance private address through stack outputs or service discovery and run the health request from a host associated with the approved client security group. Confirm that an unapproved test source cannot connect and that the application exposes only the intended health/404 behavior.
Why this step matters
Local health proves process state but not network policy. A request from the approved client tier exercises routing, SG source relationship, listener, and application. A negative test from an unapproved SG proves ingress is not accidentally broader. The root endpoint returns 404 to minimize accidental content; a real application would add authenticated routes and TLS termination.
What to understand
Use VPC Flow Logs to correlate accepted and rejected paths.
Do not expose a public test endpoint to simplify validation.
Test availability over both subnets/endpoints in the multi-instance next stage.
System changes
- Sends safe HTTP GET requests only.
Syntax explained
--connect-timeout 3 --max-time 5- Bounds a failed network/application check.
--fail- Returns non-zero for HTTP errors on the health path.
curl --fail --silent --show-error --connect-timeout 3 --max-time 5 http://PRIVATE_ADDRESS_REDACTED:8080/healthz && curl --silent --output /dev/null --write-out '%{http_code}\n' http://PRIVATE_ADDRESS_REDACTED:8080/healthy 404
Checkpoint: Only approved private clients reach the healthy service
aws ec2 describe-security-group-rules --filters Name=group-id,Values="$INSTANCE_SG" --query 'SecurityGroupRules[?IsEgress==`false`].[FromPort,ToPort,ReferencedGroupInfo.GroupId]' --output tableContinue whenApproved client gets 200 healthy, root behavior matches design, unapproved source is denied, and only client SG port 8080 ingress exists.
Stop whenPublic/unapproved source connects, health differs, listener exposes extra content, or logs contain private data.
If this step fails
Approved client times out but SSM local curl succeeds.
Likely causeSG reference, routing, NACL, client subnet, or service bind path is wrong.
Inspect flow logs, routes, NACLs, SG rule IDs, and listener address.
ResolutionCorrect only the intended private path through a change set.
Security notes
- Health endpoints should reveal minimal status, not version, dependency, or secret details.
Alternatives
- Use an internal ALB target health check and authenticated synthetic monitor.
Stop conditions
- Stop acceptance until positive and negative network tests pass.
verification
Correlate CloudTrail, stack drift, logs, and security findings
Confirm CloudTrail records stack, EC2, IAM, endpoint, SG, and Session Manager actions; run stack drift; verify session/web logs and create alerts for public exposure, metadata weakening, SG changes, role changes, SSM Offline, disk/CPU, and service health.
Why this step matters
CloudFormation ownership is credible only while drift remains detected and remediated. CloudTrail records control-plane calls but data events and session content require separate configuration. Monitoring must cover both availability and hardening regression: public addresses/routes, SG rule changes, metadata options, volume encryption, role/policy changes, endpoint state, agent connectivity, service errors, and instance resource pressure. An alert without owner and runbook is not a control.
What to understand
Verify organization trail coverage, region, encryption, retention, validation, and delivery alarms.
Enable AWS Config/Access Analyzer/Security Hub according to account standards.
Store session logs in a separate protected log account where possible.
System changes
- Starts read-only drift detection and audit queries.
Syntax explained
detect-stack-drift- Compares supported live resource properties with stack expectations.
lookup-events- Queries recent management events; not a replacement for retained trail logs.
Values stay on this page and are never sent or saved.
DETECTION_ID=$(aws cloudformation detect-stack-drift --stack-name {{stackName}} --query StackDriftDetectionId --output text) && aws cloudformation wait stack-drift-detection-complete --stack-drift-detection-id "$DETECTION_ID" && aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id "$DETECTION_ID" --query '[DetectionStatus,StackDriftStatus,DriftedStackResourceCount]' --output table && aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue={{stackName}} --max-results 10 --output table--------------------------------------- | DescribeStackDriftDetectionStatus | +-----------+----------+-------------+ | DETECTION_COMPLETE | IN_SYNC | 0 | +-----------+----------+-------------+ LookupEvents: CreateChangeSet, ExecuteChangeSet, CreateStack recorded
Checkpoint: Ownership and audit remain observable
aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].[StackStatus,EnableTerminationProtection]' --output tableContinue whenStack IN_SYNC, termination protection policy accepted, CloudTrail session-lifecycle logs present, compatible shell-session content logging tested, alerts exercised, and owners/runbooks assigned.
Stop whenDrift, missing trail/session evidence, disabled alert, or uncontrolled policy/network change remains.
If this step fails
Stack reports NOT_CHECKED or UNKNOWN drift.
Likely causeUnsupported resources, detection failure, or insufficient permissions.
Inspect detection status reason and per-resource drift support.
ResolutionAdd direct invariant checks for unsupported properties and repair detection permissions.
Security notes
- Central logs contain operator identity and host activity; restrict and retain appropriately.
Alternatives
- Use AWS Config conformance packs plus custom controls for continuous validation.
Stop conditions
- Stop release closure until audit and monitoring gaps have owners.
warning
Rehearse safe updates, CloudFormation rollback, and protected teardown
Create an update change set for a reversible web configuration/image change, reject unexplained replacements, execute, and verify. On failure let CloudFormation roll back and inspect first-cause events. Teardown requires data retention, dependency checks, and explicit removal of instance termination protection before stack deletion.
Why this step matters
Updates can replace an instance or endpoint even when a template diff looks small. The change set's Replacement property and application data model decide safety. CloudFormation rollback restores declared infrastructure where possible, but it cannot reverse external side effects or data migrations. Termination protection intentionally complicates delete/replace and must be toggled only under approved recovery/teardown. A single host causes downtime during replacement; production should evolve to a multi-instance immutable rollout.
What to understand
Use a harmless reversible update and record before/after AMI, package, metadata, SG, role, and health state.
Never skip resources in continue-update-rollback without understanding resulting drift.
Before delete, export evidence, revoke sessions, detach retained data, and verify no endpoint/network dependency is shared.
System changes
- Creates an update preview; executing later may replace or reconfigure resources.
Syntax explained
change-set-type UPDATE- Calculates changes against the existing stack.
Replacement Conditional- Replacement depends on actual property delta and needs detailed review.
Values stay on this page and are never sent or saved.
aws cloudformation create-change-set --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} --change-set-type UPDATE --template-body file://infrastructure/private-web.yml --parameters file://infrastructure/rollback-drill-parameters.json --capabilities CAPABILITY_NAMED_IAM --query Id --output text && aws cloudformation wait change-set-create-complete --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} && aws cloudformation describe-change-set --stack-name {{stackName}} --change-set-name rollback-drill-{{releaseVersion}} --query 'Changes[].ResourceChange.[Action,LogicalResourceId,Replacement]' --output tablearn:aws:cloudformation:eu-central-1:REDACTED:changeSet/rollback-drill/REDACTED --------------------------------------------------------- | DescribeChangeSet | +--------+------------------------+----------------------+ | Modify | WebInstance | Conditional | +--------+------------------------+----------------------+
Checkpoint: Update and rollback preserve private/hardening invariants
aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output textContinue whenStack returns UPDATE_COMPLETE or UPDATE_ROLLBACK_COMPLETE, then all SSM/network/IMDS/EBS/web/audit checks pass.
Stop whenReplacement is unexplained, data retained on root disk, rollback cannot restore, termination protection bypass is improvised, or hardening regresses.
If this step fails
Delete stack hangs on protected instance.
Likely causeDisableApiTermination remains true by design.
Confirm stack delete intent, retention evidence, and physical instance identity.
ResolutionDisable termination protection through an approved update/API action, record it, then retry controlled deletion.
Security notes
- Teardown must revoke instance/session authority and inspect retained snapshots/logs.
Alternatives
- Use blue/green ASG replacement and traffic shift rather than in-place update.
Stop conditions
- Stop any rollback/delete that could remove unbacked data or shared network dependencies.
Finish line
Verification checklist
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[0].Instances[0].[PublicIpAddress,KeyName]' --output textBoth values are None; no port 22, internet route, or broad network rule exists.aws ssm describe-instance-information --filters Key=InstanceIds,Values="$INSTANCE_ID" --query 'InstanceInformationList[0].PingStatus' --output textInstance is Online through private endpoints; session lifecycle is attributable in CloudTrail, and compatible shell-session content logging is verified without implying coverage for SSH or port forwarding.aws ec2 describe-instance-metadata-options --instance-ids "$INSTANCE_ID" && aws ec2 describe-volumes --filters Name=attachment.instance-id,Values="$INSTANCE_ID"IMDS tokens required, hop limit one, tags disabled, and every attached EBS volume encrypted.curl --fail --connect-timeout 3 --max-time 5 http://PRIVATE_ADDRESS_REDACTED:8080/healthzApproved client tier receives healthy while unapproved sources are denied.aws cloudformation describe-stacks --stack-name {{stackName}} --query 'Stacks[0].StackStatus' --output textStack is complete/in sync, CloudTrail and session evidence exist, and alarms are active.Recovery guidance
Common problems and safe checks
The instance never becomes a Systems Manager managed node.
Likely causeSSM Agent is absent/stopped, role policy is incomplete, endpoint DNS/security groups are wrong, time is skewed, or the subnet cannot reach required services.
Inspect stack events, endpoint state/private DNS, instance profile association, and SSM inventory.Use EC2 serial console only under approved break-glass access to read agent logs.
ResolutionCorrect the image, explicit policy, DNS, endpoint, or security-group configuration through a new change set; do not open SSH or internet egress.
Interface endpoint creation fails.
Likely causeService unavailable in region/AZ, private DNS conflict, subnet address exhaustion, quota, or duplicate endpoint exists.
Describe endpoint services, subnets, existing endpoints, DNS settings, and quotas.
ResolutionChoose supported subnets/services or adopt the existing reviewed endpoint through a separate architecture change.
dnf cannot install the web package.
Likely causeS3 gateway route/egress, DNS, repository metadata, image release, or package signature is unavailable.
Read bounded cloud-init/dnf logs through SSM or console and inspect S3 endpoint routes.
ResolutionRepair private package connectivity or use a pre-baked golden image; do not add unrestricted NAT/internet egress.
IMDS request without a token succeeds.
Likely causeLaunch template MetadataOptions did not apply, instance was replaced from another template, or HttpTokens remains optional.
Describe instance metadata options and launch template version.
ResolutionUpdate through a reviewed change set to HttpTokens required and replace/restart only as documented.
Instance has a public IPv4 address.
Likely causeSubnet mapPublicIpOnLaunch, network interface AssociatePublicIpAddress, or manual attachment overrides private design.
Describe subnet and instance network interfaces and stack drift.
ResolutionStop service acceptance, remove exposure through CloudFormation, rotate any credentials exposed during the interval, and investigate drift.
Web health works locally but not from the client tier.
Likely causeSource security group, port, routing, NACL, listener, DNS, or service bind address differs.
Verify service listener, instance/client SG references, route tables, NACLs, and VPC flow logs.
ResolutionCorrect the narrow intended path; never replace source-SG ingress with broad CIDR access.
CloudFormation change set shows instance replacement.
Likely causeImmutable subnet, image, storage, or launch property changed.
Inspect Replacement and Scope for the exact resource, plus data/availability dependencies.
ResolutionUse a tested replacement/traffic migration or reject the change; never execute an unexplained replacement.
Stack enters UPDATE_ROLLBACK_FAILED.
Likely causeA resource cannot restore, was manually changed, dependency is unavailable, or termination protection blocks replacement/deletion.
Preserve stack events and identify the first failed logical resource.
ResolutionUse continue-update-rollback with explicit resources only after recovery review; avoid manual divergence.
Session starts but no audit log appears.
Likely causeSession Manager preferences, log group/KMS permission, region, or CloudWatch endpoint is missing.
Inspect Session Manager preferences, explicit IAM actions, endpoint, log stream, and CloudTrail StartSession event.
ResolutionBlock routine sessions until logging is repaired, except documented emergency access with compensating evidence.
EBS volume reports unencrypted.
Likely causeBlockDeviceMappings omitted Encrypted, an attached data volume is outside the stack, or account default encryption is off.
Describe every attached volume and launch template block mapping.
ResolutionSnapshot/copy with encryption and replace through a tested migration; do not copy sensitive data to plaintext storage.
Security group drift adds broad egress.
Likely causeManual rule, another stack/tool, or default SG behavior changed the resource.
Run stack drift, describe rules with identifiers, and correlate CloudTrail authorization events.
ResolutionRevoke unauthorized rule under incident control, rotate exposed credentials if needed, and restore stack ownership.
Web service restarts repeatedly after boot.
Likely causeUser data/package/config, port conflict, permissions, resource pressure, or dependency failure.
Read bounded systemd, nginx, cloud-init, and kernel logs through audited SSM.
ResolutionReplace with a corrected reviewed image/template or roll back the stack update; do not edit production manually without recording drift.
Recovery
Rollback
Use a reviewed CloudFormation update change set to restore the previous image/template. Preserve first-cause events and let automatic rollback complete. The host is replaceable and contains no durable data. If teardown is required, verify retention/dependencies, disable instance API termination protection deliberately, delete through the stack, and confirm endpoints, ENIs, roles, volumes, and logs follow their retention policy.
- Freeze further change sets and capture stack/change-set IDs, events, template digest, resolved AMI, instance/network/role state, SSM status, health, and CloudTrail.
- Keep the host unreachable from clients if health or security fails; do not open SSH/public egress for repair.
- Let CREATE/UPDATE rollback complete and identify the first failed logical resource.
- If automatic rollback succeeds, rerun every private-network, SSM, IMDSv2, EBS, web, log, and drift check.
- If rollback fails, use continue-update-rollback only with reviewed resource skips and a plan to remove resulting drift.
- For a forward/previous-image update, create and review a new change set and reject unexplained replacements or policy expansion.
- Before deletion, export evidence, confirm no durable data/shared endpoint, terminate sessions, and satisfy retention policy.
- Disable API termination protection only for the exact instance under approved teardown, then delete the stack.
- Confirm no orphan ENI, endpoint, role/profile, SG, route, volume, or unexpected retained log remains.
Evidence