Adding support for EFS mounts.

Changelog:
* Upgraded CDK version to support EFS usage
* Upgraded Fargate PlatformVersion to support EFS mounts
* Refacored RDS contruct as per new CDK
* Created a new LogGroup for OnDemand DagTasks
* Added TAG for stack, to track resources belonging to this setup
* Updated sample DAG to utilize EFS. Tasks Odd and Even will publish to EFS and Numbers will read from EFS
* Now you can see logs from OnDemand tasks on Airflow UI, once task run finishes
This commit is contained in:
Chaithanya Maisagoni
2020-12-08 11:58:26 -08:00
parent 91438606d9
commit 7814377342
12 changed files with 2568 additions and 576 deletions

View File

@@ -28,6 +28,7 @@ def get_ecs_operator_args(taskDefinitionName, taskContainerName, entryFile, para
launch_type="FARGATE",
# The name of your task as defined in ECS
task_definition=taskDefinitionName,
platform_version="1.4.0",
# The name of your ECS cluster
cluster=os.environ['CLUSTER'],
network_configuration={
@@ -44,7 +45,9 @@ def get_ecs_operator_args(taskDefinitionName, taskContainerName, entryFile, para
'command': ["python", entryFile, param]
}
]
}
},
awslogs_group="FarFlowDagTaskLogs",
awslogs_stream_prefix="FarFlowDagTaskLogging/"+taskContainerName
)
oddTaskConfig = {