An AWS VPC is still just a network

By Chris Bywaters
Managing Partner, Operations and Technology

Over the course of several weeks, myself and one of my developers had been wrestling on and off with an RDS connectivity issue. We had worked through the various issues and nuances of migrating a MYSQL database into RDS instance.. Using just the configuration options in the RDS console, we quickly achieved success migrating and testing connectivity with the Database configuration option for Public accessibility being set both to "yes" and "no" and doing virtually nothing with Security Groups. However, our need was to make the database available via our VPN as well as externally to specific IP addresses. Despite our efforts we only seemed to have luck with connectivity in the VPC when we had the Public accessibility set to "no" and external connectivity only worked when we set it to "yes" after we implemented the appropriate Security Group restriction to prevent the database from being wide open to the Internet. We spent quite a bit of time attempting to resolve the issue by adjusting the Security Groups for the VPC, the subnets, the routing rules, and moving the instance to the same availability zone as our EC2 instances. None of this resolved out issue and we were positive the the Security Groups were correct.

Finally accepting that there was no single post online that was going to tell me exactly how to fix our issue, I returned to the basics. I knew that the Endpoint for the RDS database should resolve to an internal/private IP address from a system on our VPC and to public IP address on an external system. I decided to test the name resolution. I found out that inside the VPC the Endpoint was resolving to the public IP when we set Public accessibility to "yes" and to the internal only when this option was set to "no". All AWS documentation indicates that systems inside the VPC should always resolve the Endpoint to a private IP address. So our issue was name resolution. After examining the node types of the both the Ubuntu EC2 and the Windows EC2 instances I realized something that should have been obvious. All of the servers were pulling their DNS and other name resolution settings from the DHCP Options Set assigned to the VPC. I had not set this up so I was not aware that this was in place. So what was wrong with our DHCP Options Set? Well I am not sure if someone had overwritten the default or way back when it was configured the default did not include a setting that configures the EC2's to use and Amazon Provided DNS service. So in the end the solution was so simple. I created a new DHCP Options Set which has the AmazonProvidedDNS setting entered as the first item in the list of domain-name-servers. This had no negative impact on the Windows/CRM systems that coexisted with the LAMP stack and we were finally able to get the MYSQL RDS database configured to be accessible via our VPN/VPC and a group of specific external IP addresses

So in short if you are having issues connecting to an RDS Endpoint inside of you VPC you should verify that you have the DHCP Option Set configured to leverage AmazonProvidedDNS .

Below are two documents/pages at Amazon that should assist you with getting name resolution working in your VPC:

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html

Article last updated on 4/1/2019