Check out strongswan vpn… I’m not sure if they have a windows client available for the server software, but you should be able to install Linux inside a vm and use that as a fallback option (assuming you’re comfortable with linux).
Put pfsense on any router or virtual machine with a dedicated interface to the internet (eg with a public ip) and use the AWS site to site wizard available as an additional package.
Once package is installed you just feed it an IAM credential and pick which VPC to tunnel to in the list. It’ll set everything up in AWS for you, at which point you can inspect what it built.