Use Case:-
Our company has an SFTP server in the lower environment, but we don't have regular traffic on it. Our QA team only uses it once every two weeks during production release testing. If you create an AWS SFTP server, you will be charged $0.30 per hour, and AWS will continue to charge you until you delete the server. When attempting to stop the SFTP server, you will receive the following message.
Our expenditure on SFTP servers in lower environments is nearly $750 per month, despite our infrequent usage throughout the entire month. Can we tear up/down during off hours or can we create SFTP sever on Demand? Here is what I came up with a solution.
The Architecture:
To make it simple I created a simple Step Function, we can extend this functionality if we want to call Step Function from API Gateway or create an EventBridge schedule.
Create SFTP serve step function:-
Delete SFTP server step function:-
Steps:-
Configured EventBridge Scheduler.
Create a Step Function State Machine using AWS ASL language.
Add user information necessary for SFTP user creation, including the user's ID, S3 bucket path, and SSH public key, to the static data stored in DynamoDB. Additionally, once the SFTP server is successfully created, store its ID in this table. This id will use to delete the SFTP server.
You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.
CDK Deploy:-
1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>
SAM Deploy:-
1) cd sam
2) sam deploy --stack-name SFTPAppStack --capabilities CAPABILITY_NAMED_IAM --guided --profile <profile name>
Populate Static Data in DynamoDB:-
1) cd dynamodb
2) ./dynamodb.sh
You will see the below records in the DynamoDB table.
you can use this link to generate your private key and public key. Modify your public key in the DynamoDB script.
Populate Dummy Data in S3:-
1) cd s3
2) ./s3.sh
Understand State Machine:-
I am using the AWS SDK integration API to create an SFTP server with the help of the endpoint "arn:aws:states:::aws-sdk:transfer:createServer". There are various parameters available for creating the server, but it's important to ensure the security aspect is verified if you plan to create an SFTP server in your account.
"CreateServer": {
"Type": "Task",
"Next": "Save ServerId",
"Parameters": {
"Domain": "S3",
"EndpointType": "PUBLIC",
"IdentityProviderType": "SERVICE_MANAGED",
"LoggingRole": "${SFTPBaseRole}",
"Protocols": [
"SFTP"
],
"SecurityPolicyName": "TransferSecurityPolicy-2018-11"
},
"Resource": "arn:aws:states:::aws-sdk:transfer:createServer"
},
Above execution will return sever-id which I am saving into the DynamoDB table. This server-id will be used to delete the SFTP server. Here is the code snippet to save server-id into the DynamoDB table using SDK endpoint "arn:aws:states:::dynamodb:putItem"
"Save ServerId": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "${SFTPTable}",
"Item": {
"PK": {
"S": "SFTP"
},
"SK": {
"S": "ServerId"
},
"ServerId": {
"S.$": "$.ServerId"
}
}
},
"Next": "Get SFTP User",
"ResultPath": null
}
Now I am fetching all SFTP users from the DynamoDB table. This users list we can iterate using Map and pass it to endpoint "arn:aws:states:::awssdk:transfer:createUser".
"Get SFTP User": {
"Type": "Task",
"Parameters": {
"TableName": "${SFTPTable}",
"KeyConditionExpression": "PK = :v1 AND begins_with(SK,:v2)",
"ExpressionAttributeValues": {
":v1": {
"S": "SFTP"
},
":v2": {
"S": "USER"
}
}
},
"Resource": "arn:aws:states:::aws-sdk:dynamodb:query",
"Next": "Iterate User",
"ResultPath": "$.context"
Iterate all users list using the map.
"Iterate User": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Create SFTP User",
"States": {
"Create SFTP User": {
"Type": "Task",
"End": true,
"Parameters": {
"ServerId.$": "$.ItemSelector.ServerId",
"HomeDirectory.$": "$.ItemSelector.Value.S3Path.S",
"HomeDirectoryType": "PATH",
"Role": "${SFTPBaseRole}",
"SshPublicKeyBody.$":"$.ItemSelector.Value.SSHPublicKey.S",
"UserName.$": "States.ArrayGetItem(States.StringSplit($.ItemSelector.Value.SK.S, '#'), 1)"
},
"Resource": "arn:aws:states:::aws-sdk:transfer:createUser"
}
}
},
"End": true,
"ItemsPath": "$.context.Items",
"ItemSelector": {
"ItemSelector": {
"ServerId.$": "$.ServerId",
"Value.$": "$$.Map.Item.Value"
}
}
}
The $$.Map.Item.Value.Value context object contains the value of each data item. You can use $$.Map.Item.Value if you want to add/modify JSON elements in iteration. You can see I am adding the new attribute "ServerId.$": "$.ServerId"
Login to SFTP Server:-
You should be able to log in using the following hostname. I am using the us-west-2 region. Modify according to your region. Use any user you have created with your private key.
<server-id>.server.transfer.us-west-2.amazonaws.com
Delete SFTP Server:-
Getting ServerId from DynamoDB and passing this value to endpoint "arn:aws:states:::aws-sdk:transfer:deleteServer"
{
"Comment": "A description of my state machine",
"StartAt": "Get SFTP ServerId",
"States": {
"Get SFTP ServerId": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "${SFTPTable}",
"Key": {
"PK": {
"S": "SFTP"
},
"SK": {
"S": "ServerId"
}
}
},
"Next": "DeleteServer",
"OutputPath": "$.Item"
},
"DeleteServer": {
"Type": "Task",
"End": true,
"Parameters": {
"ServerId.$": "$.ServerId.S"
},
"Resource": "arn:aws:states:::aws-sdk:transfer:deleteServer"
}
}
}
Cleanup:
sam delete --stack-name SFTPStack --profile <your profile name>
OR
cd cdk
cdk destroy --profile <your profile name>
Wrap-Up:-
By creating or deleting a server during off hours, we can observe how much money can be saved. At my company, we have made the decision to generate an SFTP server on an as-needed basis. I'm confident that our costs will be less than $10 per month, in contrast to the $750 expense for a non-production environment. As I mentioned earlier, this post is not about how to create a secured SFTP server, please ensure the security aspect if you are creating an SFTP server in your account. Hope you like this post and helps you to reduce your AWS $$$ bill.