I spoke with a customer that mentioned they were moving an Exadata rack from one datacenter to another and they had a few questions about what needed to change in order to complete the move. The customer wanted to make the move as much of a "lift and shift" as possible, avoiding the need to rebuild the cluster upon power up at the new datacenter. This means that the hostnames themselves could not change. We knew that IP addresses would have to change, but the customer also has different DNS domain names for each datacenter. For example, with Exadata racks in Atlanta and Dallas, the SCAN hostnames might be:
exa1-scan.atl.client.com
exa2-scan.dal.client.com
The relocation happened to be on a very tight timeline, and the application team requested that the original domain name be kept after moving the Exadata, since they did not have time to test changing their applications. That raised the question on whether we should change the domain name at all, or just leave this Exadata as the misfit with the wrong domain. I raised the point that they could easily update the DNS records in the old domain to point to the new IPs, and create records in the new domain. Unfortunately, this means 2 sets of DNS records that exist for a period of time, but it takes care of the permanent state as well as keeps the application team running as they are today.
The next question was about how the SCAN would reply to a client that connects with a fully qualified domain name (FQDN) that is using the old domain. I wanted to do a test to see what would happen. As we know, a datbase client first connects to the SCAN listener, and is then directed to the appropriate database instance via a local listener running a on a VIP. The concern of the customer was that an application querying the SCAN with one domain name may receive a result that it can't resolve, breaking connectivity after the move.
First, it's worth noting that (on Exadata), the VIP hostnames are stored in the cluster registry as an FQDN, whereas the SCAN is not:
[root@enkx6db01 ~]# srvctl config vip -node enkx6db01
VIP exists: network number 1, hosting node enkx6db01
VIP Name: enkx601-vip.enkitec.local
VIP IPv4 Address: 10.9.238.68
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
[root@enkx6db01 ~]# srvctl config scan
SCAN name: enkx6-scan, Network: 1
Subnet IPv4: 10.9.238.64/255.255.255.192/bondeth0, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 10.9.238.70
SCAN VIP is enabled.
SCAN 2 IPv4 VIP: 10.9.238.71
SCAN VIP is enabled.
SCAN 3 IPv4 VIP: 10.9.238.72
SCAN VIP is enabled.
I decided to test out the scenario by removing any domain names from my client's search path, so a non-fully qualified hostname would cause the connection to fail. As I expected, the database client was able to connect successfully. I ran a test to a database named x619 on the enkx6-scan cluster.
That left the question - what exactly does the SCAN return when the client connects? I pulled out my old friend, tcpdump, and ran the following command to capture all packets back and forth between my client and the SCAN listeners:
# tshark -i eth0 -V host 10.9.238.70 or 10.9.238.71 or 10.9.238.72
We ended up with the following frame of data, sent from the SCAN to my client. Output is abbreviated with "..." to clarify when some output has been removed.
Frame 7: 325 bytes on wire (2600 bits), 325 bytes captured (2600 bits) on interface 0
Interface id: 0
WTAP_ENCAP: 1
Arrival Time: Apr 5, 2021 21:22:05.909223555 CDT
...
Internet Protocol Version 4, Src: 10.9.238.70 (10.9.238.70), Dst: 10.9.237.131 (10.9.237.131)
...
Transmission Control Protocol, Src Port: ncube-lm (1521), Dst Port: 46743 (46743), Seq: 11, Ack: 257, Len: 271
Source port: ncube-lm (1521)
Destination port: 46743 (46743)
[Stream index: 0]
Sequence number: 11 (relative sequence number)
[Next sequence number: 282 (relative sequence number)]
Acknowledgment number: 257 (relative ack number)
Header length: 20 bytes
Flags: 0x018 (PSH, ACK)
...
Transparent Network Substrate Protocol
Packet Length: 271
Packet Checksum: 0x000a
Packet Type: Data (6)
Reserved Byte: 00
Header Checksum: 0x0000
Data
Data Flag: 0x0040
.... .... .... ...0 = Send Token: False
.... .... .... ..0. = Request Confirmation: False
.... .... .... .0.. = Confirmation: False
.... .... .... 0... = Reserved: False
.... .... ..0. .... = More Data to Come: False
.... .... .1.. .... = End of File: True
.... .... 0... .... = Do Immediate Confirmation: False
.... ...0 .... .... = Request To Send: False
.... ..0. .... .... = Send NT Trailer: False
Data (261 bytes)
0000 28 41 44 44 52 45 53 53 3d 28 50 52 4f 54 4f 43 (ADDRESS=(PROTOC
0010 4f 4c 3d 54 43 50 29 28 48 4f 53 54 3d 31 30 2e OL=TCP)(HOST=10. <- IP ADDR
0020 39 2e 32 33 38 2e 36 38 29 28 50 4f 52 54 3d 31 9.238.68)(PORT=1 <- IP ADDR
0030 35 32 31 29 29 00 28 44 45 53 43 52 49 50 54 49 521)).(DESCRIPTI
0040 4f 4e 3d 28 43 4f 4e 4e 45 43 54 5f 44 41 54 41 ON=(CONNECT_DATA
0050 3d 28 53 45 52 56 45 52 3d 44 45 44 49 43 41 54 =(SERVER=DEDICAT
0060 45 44 29 28 53 45 52 56 49 43 45 5f 4e 41 4d 45 ED)(SERVICE_NAME
0070 3d 78 36 31 39 29 28 43 49 44 3d 28 50 52 4f 47 =x619)(CID=(PROG
0080 52 41 4d 3d 73 71 6c 70 6c 75 73 29 28 48 4f 53 RAM=sqlplus)(HOS
0090 54 3d 65 6e 6b 70 65 78 61 63 68 6b 2e 65 6e 6b T=enkpclient.enk
00a0 69 74 65 63 2e 6c 6f 63 61 6c 29 28 55 53 45 52 itec.local)(USER
00b0 3d 6f 72 61 63 6c 65 29 29 28 49 4e 53 54 41 4e =oracle))(INSTAN
00c0 43 45 5f 4e 41 4d 45 3d 78 36 31 39 31 29 29 28 CE_NAME=x6191))( <- INSTANCE
00d0 41 44 44 52 45 53 53 3d 28 50 52 4f 54 4f 43 4f ADDRESS=(PROTOCO
00e0 4c 3d 54 43 50 29 28 48 4f 53 54 3d 31 30 2e 39 L=TCP)(HOST=10.9
00f0 2e 32 33 38 2e 37 30 29 28 50 4f 52 54 3d 31 35 .238.70)(PORT=15
0100 32 31 29 29 29 21)))
Data: 28414444524553533d2850524f544f434f4c3d5443502928...
[Length: 261] Sure enough, the SCAN reported back with the IP address of the VIP. In this case, the customer could easily remove all of the DNS records in the old domain, leaving a CNAME that points the old SCAN hostname to the new one. In the end, they opted to update the DNS records, just in case anyone misses the domain change.
Wonderful explanation !!