ECS `RunTask` needs public IP to access env file

Learning how to use ECS tasks to run some cron jobs has been an opaque journey to say the least.

I knew my TaskDefinition was fine, because my server was running, but I wanted to use that same definition with a different container command and schedule its execution for housekeeping tasks.

I started with creating an EventBridge schedule, which seemed straight forward enough. But as soon as I created it I was puzzled that there didn't seem to be a way to trigger it for testing nor was there any kind of information on whether or not the last invocation had happened and succeeded.

Incidentally, until I devise a comment strategy for this blog, hit me up on mastodon in particular this post about trying to figure EventBridge monitoring, if you have anything to add to this topic.

Checking the next day, it didn't seem like the cron job had run and there was nothing in the logs, since that only emits if it actually got to the running state. After some digging, I found the RunTask event in CloudTrail, but trying to look up the task, it was MISSING. Apparently completed tasks are only stored for a short period.

Trying to reduce the surface area of this problem, I decided to run the task with command override directly in ECS, which resulted in:

"stoppedReason": "ResourceInitializationError: failed to download env files: file download command: non empty error stream: service call has been retried 5 time(s): RequestCanceled: request context canceled caused by: context deadline exceeded"

I currently keep my task environment vars in a file in a private S3 bucket. Could have taken this as a hint to switch to ParameterStore instead, but I was determined to figure this out, since it worked for the server.

I should have just googled that error right then and there, since when I eventually did that, I found out what took me way too long to devise on my own. Instead, I ran the task a number of times, starting with the server configuration and backing out things I didn't think were needed one at a time until I discovered that the task needs a public IP to access the env file , i.e.:

"NetworkConfiguration": {
    "awsvpcConfiguration": {
      "Subnets": [
      ],
      "SecurityGroups": [
      ],
      "AssignPublicIp": "ENABLED"
    }
  }

Oh, and as a final source of confusion, when trying to use a CapacityProviderStrategy of FARGATE in EventBridge ENABLED was not an option in the console. Only after switching to LaunchType FARGATE did that become available.