How Kubernetes Ingress Works with VMware Avi Load Balancer

Last updated: September 2026

In this lesson, we will learn how Kubernetes Ingress Works with VMware Avi Load Balancer

Kubernetes Ingress can be confusing when you first start working with it, especially when it is integrated with a load balancer such as VMware Avi Load Balancer.

You may see something like:

Ingress
↓
Avi Load Balancer
↓
Virtual Service
↓
Pool
↓
Pod IPs
↓
Application

But what exactly happens between these components?

What is the VIP?

Where does the VIP come from?

Why does the Ingress show an IP address even though you didn’t put an IP address in the Ingress YAML?

What is an Avi Virtual Service?

What is a Pool?

Why does the Pool contain Pod IP addresses?

Why does Avi show ports 80 and 443 even though the Ingress YAML doesn’t explicitly define those ports?

What happens when DNS is not configured?

And how can you test the application when DNS is broken?

This article walks through these questions step by step using a practical Kubernetes (VKS) + VMware Avi example.

Understanding the Subject Matter

What is Kubernetes Ingress?

A Kubernetes Ingress is an API object used to define how HTTP/HTTPS traffic should enter a Kubernetes environment and reach applications running inside the cluster.

For example, suppose we have an application called:

fintechai-tekneed

and we want users to access it through:

prod-tkg.fintechapi.tekneed

We could create an Ingress similar to:

apiVersion: networking.k8s.io/v1
kind: Ingress

metadata:
  name: fintechai-tekneed
  namespace: fintech-production

spec:
  ingressClassName: avi-lb

  rules:
  - host: prod-tkg.fintechapi.tekneed
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: fintechai-tekneed
            port:
              number: 30084

  tls:
  - hosts:
    - prod-tkg.fintechapi.tekneed
    secretName: wildcard-fintechapi-2026-cert

There are several important things here.

We have:

ingressClassName: avi-lb

This tells Kubernetes which Ingress controller is responsible for this Ingress.

In this environment, that controller is Avi Kubernetes Operator (AKO).

We also have:

host: prod-tkg.fintechapi.tekneed

This is the hostname users will use.

And:

backend:
  service:
    name: fintechai-tekneed
    port:
      number: 30084

This tells the Ingress where traffic should go inside Kubernetes, which is usually the service name (ClusterIP service type most times)

Finally:

tls:
- hosts:
  - prod-tkg.fintechapi.tekneed
  secretName: wildcard-fintechapi-2026-cert

This tells the Ingress that HTTPS/TLS is involved and identifies the Kubernetes TLS secret containing the certificate.

Is Ingress Layer 7?

Yes.

This is one of the most important things to understand.

Ingress is primarily concerned with HTTP/HTTPS traffic, which is Layer 7 of the OSI model.

For example, an Ingress can make decisions based on:

Hostname
Path
HTTP/HTTPS
TLS

For example:

prod-tkg.fintechapi.tekneed/api

could be sent to one application while:

prod-tkg.fintechapi.tekneed/payment

could be sent to another application.

That is Layer 7 routing.

A traditional Layer 4 load balancer mainly cares about things such as:

IP address
TCP
UDP
Port

An Avi Virtual Service created from an Ingress can therefore perform Layer 7 processing.

You may consequently see Avi Virtual Services with names containing:

L7-dedicated

The L7 refers to Layer 7.

The Kubernetes Service Behind the Ingress

Before understanding Avi, we need to understand the Kubernetes Service.

Suppose we run:

kubectl get svc fintechai-tekneed -n fintech-production

We might get:

NAME               TYPE        CLUSTER-IP       PORT(S)
fintechai-tekneed  ClusterIP   10.250.241.202   30084/TCP,4317/TCP

At first glance, this can be confusing.

Why does the Service have two ports?

Let’s look at the YAML:

apiVersion: v1
kind: Service

metadata:
  name: fintechai-tekneed
  namespace: fintech-production

spec:
  clusterIP: 10.250.241.202

  ports:

  - name: 8000-tcp
    port: 30084
    protocol: TCP
    targetPort: 8000

  - name: 4317-tcp
    port: 4317
    protocol: TCP
    targetPort: 4317

  selector:
    app: fintechai-tekneed

  type: ClusterIP

There are two Service ports:

30084 → 8000
4317  → 4317

These are not necessarily two different applications.

They are simply two different ports exposed by the same Kubernetes Service.

Understanding `port` and `targetPort`

This is extremely important.

Consider:

port: 30084
targetPort: 8000

This means:

Client
  |
  | TCP 30084
  ↓
Kubernetes Service
  |
  | forwards to TCP 8000
  ↓
Pod

So the Service listens on:

but the application container is listening on:

For example:

Service:
10.250.241.202:30084

        ↓

Pod:
10.250.24.129:8000

This is why the Ingress can refer to:

port:
  number: 30084

The Ingress is referring to the Service port, not directly to the container port.

What About Port 4317?

Our Service also has:

- name: 4317-tcp
  port: 4317
  targetPort: 4317

So:

Service port 4317
        ↓
Pod port 4317

However, the Ingress does not necessarily use this port.

Our Ingress says:

backend:
  service:
    name: fintechai-tekneed
    port:
      number: 30084

Therefore, this particular HTTP/HTTPS Ingress route uses:

30084:

not

The fact that the Service exposes 4317 does not automatically mean that the Ingress will expose 4317.

This distinction is very important.

What Happens Behind the Service?

The Service has a selector:

selector:
  app: fintechai-tekneed

Kubernetes looks for Pods matching:

app=fintechai-tekneed

Suppose we have two Pods:

fintechai-tekneed-abc123
IP: 10.250.24.129

fintechai-tekneed-def456
IP: 10.250.33.117

Both Pods have:

app=fintechai-tekneed

Kubernetes therefore considers these Pods as endpoints for the Service.

We can see them with:

kubectl get endpoints fintechai-tekneed -n fintech-production

Or, on modern Kubernetes versions, preferably:

kubectl get endpointslice -n fintech-production

The traffic flow becomes:

Ingress
   ↓
Service
   ↓
Pod 10.250.24.129:8000

or:

Ingress
   ↓
Service
   ↓
Pod 10.250.33.117:8000

Now Introduce VMware Avi

How Kubernetes Ingress Works with VMware Avi Load Balancer

This is where the architecture becomes more interesting.

In our environment, the Ingress uses:

ingressClassName: avi-lb

This means the Avi Kubernetes integration is responsible for implementing the Ingress.

The component responsible for this integration is AKO — Avi Kubernetes Operator.

Conceptually:

Kubernetes Ingress
        |
        | watches
        ↓
       AKO
        |
        | configures
        ↓
Avi Load Balancer

AKO looks at the Kubernetes objects and configures Avi accordingly.

The VIP

Avi provides a Virtual IP, commonly called a VIP.

For our sanitized example:

VIP: 10.250.77.85

This becomes the frontend IP address for the application.

Think of the VIP as the front door.

Users don’t connect directly to the Pods.

Instead, they connect to:

prod-tkg.fintechapi.tekneed

which should resolve to:

10.250.77.85

The traffic then enters Avi.

Conceptually:

User
 |
 | https://prod-tkg.fintechapi.tekneed
 |
 ↓
DNS
 |
 | 10.250.77.85
 ↓
Avi VIP
 |
 ↓
Avi Virtual Service
 |
 ↓
Avi Pool
 |
 ↓
Kubernetes Pods

Was the VIP Manually Defined in the Ingress YAML?

This is an important point.

When you run:

kubectl get ingress fintechai-tekneed -n fintech-production -o yaml

you may see something like:

status:
  loadBalancer:
    ingress:
    - hostname: prod-tkg.fintechapi.tekneed
      ip: 10.250.77.85

It is easy to think:

“Someone manually put this IP into my Ingress YAML.”

Normally, that is not what happened.

The status section is different from the desired configuration under spec.

The important distinction is:

spec:

versus:

status:

The spec describes what you are asking Kubernetes to configure.

The status reports what has actually happened.

For example:

spec:
  ingressClassName: avi-lb

is configuration.

But:

status:
  loadBalancer:
    ingress:
    - ip: 10.250.77.85

is status information.

AKO/Avi can populate that status with the address assigned to the Ingress.

Therefore, seeing:

status:
  loadBalancer:
    ingress:
    - ip: 10.250.77.85

does not mean that the IP was manually hardcoded into the original Ingress manifest.

The Ingress Doesn’t Need to Contain the VIP

Your Ingress can simply contain:

spec:
  ingressClassName: avi-lb

and the host:

host: prod-tkg.fintechapi.tekneed

AKO then communicates with Avi and Avi allocates/configures the appropriate Virtual Service/VIP.

The resulting VIP can then appear in the Ingress status.

This is similar to saying:

“Here is what I want.”

and Avi responding:

“Okay, I created it and this is the address I assigned.”

Does the VIP Change When Avi or AKO Restarts?

Generally, a restart of AKO, an Avi Controller, or a Service Engine does not mean the VIP simply changes to another random IP.

The VIP is part of the Avi configuration.

For example:

Virtual Service
       |
       ↓
VIP
10.250.77.85

A restart does not normally mean:

10.250.77.85
       ↓
10.250.99.15

The configuration persists.

A Service Engine can restart or fail and traffic can be moved to another Service Engine while the frontend VIP remains the same.

This is one of the major purposes of the load-balancer architecture.

So DNS can safely point to the VIP:

prod-tkg.fintechapi.tekneed
             ↓
        10.250.77.85

and the VIP remains the stable frontend address.

DNS Comes Before the User Reaches the VIP

Now we arrive at the part that caused a lot of confusion during troubleshooting.

Suppose the user enters:

curl https://prod-tkg.fintechapi.tekneed/

Before curl can connect to Avi, the hostname has to be resolved.

The normal process is:

prod-tkg.fintechapi.tekneed
             |
             ↓
           DNS
             |
             ↓
       10.250.77.85
             |
             ↓
        Avi VIP

Therefore, the DNS team would normally create a DNS record similar to:

prod-tkg.fintechapi.tekneed → 10.250.77.85

This is what allows users and applications to use the hostname instead of remembering the IP address.

What Happens If DNS Is Not Configured?

Suppose we run:

nslookup prod-tkg.fintechapi.tekneed

and receive:

** server can't find prod-tkg.fintechapi.tekneed: NXDOMAIN

NXDOMAIN means that the DNS server is saying:

This hostname does not exist in the DNS namespace I searched.

Therefore:

curl https://prod-tkg.fintechapi.tekneed/

will fail because the operating system cannot translate:

prod-tkg.fintechapi.tekneed

into:

10.250.77.85

You may see:

Could not resolve host: prod-tkg.fintechapi.tekneed

At this point, you haven’t even reached Avi.

The failure is occurring at DNS resolution.

But What If the VIP Is Reachable?

This is where testing becomes useful.

Suppose:

ping 10.250.77.85

works.

You might get:

[Tekneed-victor@tekneedpkscli01 ~]$ ping 10.250.77.85

PING 10.250.77.85 (10.250.77.85) 56(84) bytes of data.
64 bytes from 10.250.77.85: icmp_seq=1 ttl=58 time=1.09 ms
64 bytes from 10.250.77.85: icmp_seq=2 ttl=58 time=1.06 ms
..........

This tells us that the IP is reachable at the network level.

You can also test HTTP:

curl http://10.250.77.85

and perhaps receive:

301 Moved Permanently

from:

Avi Vantage

An example below is what I extracted from my environment/setup

[Tekneed-victor@tekneedpkscli01 ~]$ k get ing

NAME               CLASS    HOSTS                   ADDRESS        PORTS     AGE
fintechai-tekneed  avi-lb   prod-tkg.fintechapi.tekneed   10.250.77.85   80, 443   33d


[Tekneed-victor@tekneedpkscli01 ~]$ curl 10.250.77.85

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>Avi Vantage/</center>
</body>
</html>


[Tekneed-victor@tekneedpkscli01 ~]$ curl http://10.250.77.85

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>Avi Vantage/</center>
</body>
</html>
[Tekneed-victor@tekneedpkscli01 ~]$

NB: curl 10.250.77.85 and curl http://10.250.77.85 is the same

That is significant.

It tells you that you can reach the Avi frontend IP (VIP).

But it does not necessarily prove that your application is working.

Why?

Because you’re connecting directly to the IP.

The application routing may depend on the hostname.

NOTE: When troubleshooting or testing, it is advisable to use the verbose (-v) option.

curl <endpoint> and curl -v <endpoint> connection/request itself is the same. The difference is what curl shows you.

curl http://10.250.77.85

which is same as (because its http protocol)

curl 10.250.77.85

This normally shows you only the response body just as the response shown above.

curl -v

curl -v http://10.250.77.85

-v means verbose

It shows you the details of the conversation between curl and the server, which is very good for troubleshooting/testing

[Tekneed-victor@tekneedpkscli01 ~]$ curl -v http://10.250.77.85

* Rebuilt URL to: http://10.250.77.85/
* Uses proxy env variable http_proxy == 'http://10.212.1.11:8080'
*   Trying 10.212.1.11...
* TCP_NODELAY set
* Connected to 10.212.1.11 (10.212.1.11) port 8080 (#0)


> GET http://10.250.77.85/ HTTP/1.1
> Host: 10.250.77.85
> User-Agent: curl/3.44.2
> Accept: */*
> Proxy-Connection: Keep-Alive
>


< HTTP/1.1 301 Moved Permanently
< Location: https://10.250.77.85/
< Content-Length: 185
< Content-Type: text/html
< Date: Sun, 30 Aug 2026 14:34:03 GMT
< Via: 1.1 teknsw02.tekneed.com:80 (Cisco-GMA/17.2.5-214)
< Connection: keep-alive
< Proxy-Connection: keep-alive
<


<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>Avi Vantage/</center>
</body>
</html>
* Connection #0 to host 10.212.1.11 left intact

You can see the conversation that takes place.

Think of the result as:

Our machine
    |
    | HTTP request
    v
Corporate Proxy
10.212.1.11:8080
    |
    v
10.250.77.85
    |
    v
Avi

Now let’s break it down.

1. Curl understands the URL

* Rebuilt URL to: http://10.250.77.85/

You gave curl:

http://10.250.77.85

Curl normalizes it to:

http://10.250.77.85/

Nothing interesting yet.

2. Curl sees that an HTTP proxy is configured

This is the most important line:

* Uses proxy env variable http_proxy == 'http://10.212.1.11:8080'

Your Linux machine has an environment variable called:

http_proxy

and its value is:

http://10.212.1.11:8080

Therefore, curl says:

“Since I’m making an HTTP request and an HTTP proxy is configured, I’ll send the request through the proxy.”

So instead of:

Your machine ─────────────→ 10.250.77.85

you have:

Your machine ─────────────→ 10.212.1.11:8080

3. Curl connects to the proxy

* Trying 10.212.1.11...

Curl is attempting to connect to the proxy.

Then:

* Connected to 10.212.1.11 (10.212.1.11) port 8080 (#0)

This tells us:

Your machine successfully established a TCP connection to the corporate proxy on port 8080.

It does not yet prove that 10.250.77.85 is reachable.

4. Curl sends the HTTP request to the proxy

This is very important:

> GET http://10.250.77.85/ HTTP/1.1

Curl is telling the proxy:

“I want you to get http://10.250.77.85/ for me.”

Notice that the complete URL appears in the GET request.

Then:

> Host: 10.250.77.85

This identifies the destination host.

So the communication at this point is:

Your machine
     |
     | GET http://10.250.77.85/
     v
Proxy 10.212.1.11:8080

5. The proxy returns HTTP 301

The proxy returns:

< HTTP/1.1 301 Moved Permanently

A 301 means:

“The resource you’re requesting has been redirected to another URL.”

And the next line tells us where:

< Location: https://10.250.77.85/

So the destination is saying/indicating:

HTTP
 :80
  |
  | 301 Redirect
  v
HTTPS
 :443

In other words:

“Don’t use HTTP; use HTTPS.”

6. Now look at the `Avi Vantage` part

The response body contains:

<center>Avi Vantage/</center>

This is strong evidence that the response is associated with Avi.

However, notice something important:

< Via: 1.1 teknsw02.tekneed.com:80 (Cisco-GMA/17.2.5-214)

That tells us a Cisco proxy/security device is also in the path.

Therefore, because we did not use --noproxy, this is not a clean direct test from your machine to Avi.

The safest interpretation is:

The HTTP request went through the corporate proxy, and the response indicates an Avi Vantage 301 redirect to HTTPS.

7. Finally, curl closes its connection to the proxy

* Connection #0 to host 10.212.1.11 left intact

Notice again:

10.212.1.11

That’s the proxy, not:

10.250.77.85

This reinforces that curl’s direct TCP connection was to the proxy.

So what actually happened?

The command:

curl -v http://10.250.77.85

resulted in:

              Your machine
                   |
                   | HTTP
                   ↓
           10.212.1.11:8080
          Corporate Proxy
                   |
                   | HTTP request
                   ↓
             10.250.77.85
                   |
                   ↓
                  Avi
                   |
                   | 301
                   ↓
        https://10.250.77.85/

But there’s an important distinction:

What we have proven

We have proven that:

Our machine can reach the corporate proxy.
The proxy processed the HTTP request.
The response indicates an Avi Vantage 301 Moved Permanently.
HTTP is being redirected to HTTPS.

What we have not cleanly proven

We have not performed a direct, proxy-free test from our machine to 10.250.77.85.

For that, we’d use:

curl --noproxy '*' -v http://10.250.77.85

Now the path would be:

Your machine
     |
     | HTTP :80
     | NO PROXY
     ↓
10.250.77.85
     |
     ↓
Avi

That is the test that cleanly answers:

“Can my machine directly reach the Avi VIP on HTTP port 80?”

Now let’s do that

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -v http://10.250.77.85

* Rebuilt URL to: http://10.250.77.85/
*   Trying 10.250.77.85...
* TCP_NODELAY set
* Connected to 10.250.77.85 (10.250.77.85) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.250.77.85
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html
< Content-Length: 185
< Connection: keep-alive
< Location: https://10.250.77.85/
<
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>Avi Vantage/</center>
</body>
</html>
* Connection #0 to host 10.250.77.85 left intact

Now we can see the direct reachability from our machine to Avi

Now, since our test asked us to use https, let’s do that.

[Tekneed-victor@tekneedpkscli01 ~]$ curl https://10.250.77.85

curl: (51) SSL: no alternative certificate subject name matches target host name '10.250.77.85'

[Tekneed-victor@tekneedpkscli01 ~]$ curl -v https://10.250.77.85

* Rebuilt URL to: https://10.250.77.85/
* Uses proxy env variable https_proxy == 'http://10.212.1.11:8080'
*   Trying 10.212.1.11...
* TCP_NODELAY set
* Connected to 10.212.1.11 (10.212.1.11) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to 10.250.77.85:443
> CONNECT 10.250.77.85:443 HTTP/1.1
> Host: 10.250.77.85:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/mkli/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=NG; L=Lagos; O=TekNeed; CN=*.tekneed.com
*  start date: Oct  8 00:00:00 2025 GMT
*  expire date: Oct  7 23:59:59 2026 GMT
*  subjectAltName does not match 10.250.77.85
* SSL: no alternative certificate subject name matches target host name '10.250.77.85'
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (51) SSL: no alternative certificate subject name matches target host name '10.250.77.85'

1. First command: `curl https://10.250.77.85`

The first command without (-v) is just the short error, while the second command with the (-v) shows us the whole journey, just as we have established the importance of the verbose option.

The command without -v tells us something important:

Curl got far enough to perform certificate validation – meaning there was reachability until certificate validation

But the certificate presented by the server does not match the IP address used.

To understand why, we need the verbose output.

2. Curl builds the URL

* Rebuilt URL to: https://10.250.77.85/

You entered:

curl https://10.250.77.85

Curl interprets that as:

HTTPS
Host: 10.250.77.85
Port: 443
Path: /

So conceptually:

https://10.250.77.85/
        |       |
       Host    Port 443

3. Again, the proxy is involved

This line is extremely important:

* Uses proxy env variable https_proxy == 'http://10.212.1.11:8080'

Our machine has:

https_proxy=http://10.212.1.11:8080

So curl doesn’t initially connect directly to 10.250.77.85.

It first connects to:

10.212.1.11:8080

Then:

* Trying 10.212.1.11...
* Connected to 10.212.1.11 (10.212.1.11) port 8080

So:

Your machine
     |
     | TCP connection
     v
10.212.1.11:8080
Corporate Proxy

4. Curl asks the proxy to create an HTTPS tunnel

Now we get to something different from the HTTP test.

You see:

* Establish HTTP proxy tunnel to 10.250.77.85:443

Curl sends:

> CONNECT 10.250.77.85:443 HTTP/1.1
> Host: 10.250.77.85:443

This means:

“Proxy, please establish a tunnel between you and 10.250.77.85 on TCP port 443.”

The proxy responds:

< HTTP/1.1 200 Connection established

And curl says:

* Proxy replied 200 to CONNECT request
* CONNECT phase completed!

This is important.

The proxy has allowed the HTTPS tunnel.

So at this point:

Our machine
     |
     | HTTPS tunnel through proxy
     v
10.212.1.11:8080
     |
     | TCP tunnel to :443
     v
10.250.77.85:443

The proxy has not rejected the connection.

5. TLS negotiation starts

Now we see:

* TLSv1.3 (OUT): TLS handshake, Client hello

Your machine sends a TLS ClientHello.

The server responds:

* TLSv1.3 (IN): TLS handshake, Server hello

Then:

* TLSv1.2 (IN): TLS handshake, Certificate

This is where things get really interesting.

The server at 10.250.77.85 presents a certificate.

Curl tells us:

* Server certificate:
* subject: C=NG; L=Lagos; O=TekNeed; CN=*.tekneed.com

So the certificate belongs to:

*.tekneed.com

6. But we requested the IP address

The request was:

https://10.250.77.85

Therefore curl expects the certificate to be valid for:

10.250.77.85

But the certificate says:

*.tekneed.com

Those don’t match.

That’s why curl reports:

* subjectAltName does not match 10.250.77.85

and finally:

* SSL: no alternative certificate subject name matches target host name '10.250.77.85'

Then:

curl: (51) SSL: no alternative certificate subject name matches target host name '10.250.77.85'

7. So did it reach Avi?

Yes, we have strong evidence that the HTTPS service at that VIP was reached far enough to return a TLS certificate.

The important sequence is:

CONNECT 10.250.77.85:443
        ↓
200 Connection established
        ↓
TLS ClientHello
        ↓
TLS ServerHello
        ↓
Server sends Certificate
        ↓
Certificate = *.tekneed.com
        ↓
Curl compares certificate with 10.250.77.85
        ↓
NO MATCH
        ↓
Curl terminates connection

So the failure is not:

“I can’t connect to port 443.”

The failure is:

“I connected and received a certificate, but the certificate isn’t valid for the hostname I used.”

8. This is why `curl https://IP` is not the same as using the hostname

This is the key lesson.

When you do:

curl https://10.250.77.85

curl essentially says:

“I am connecting to 10.250.77.85, so the certificate needs to identify 10.250.77.85.”

But your certificate is:

CN=*.tekneed.com

Therefore:

10.250.77.85
       ❌
*.tekneed.com

doesn’t match.

But if the intended hostname were something like:

prod-tkg.fintechapi.tekneed

then:

prod-tkg.fintechapi.tekneed
        ✅
*.tekneed.com

would match the wildcard certificate.

9. And notice something else: this test used the proxy

This is another important distinction from the test we did earlier.

We can bypass the coporate proxy by using the command,

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -vk https://10.250.77.85

* Rebuilt URL to: https://10.250.77.85/
*   Trying 10.250.77.85...
* TCP_NODELAY set
* Connected to 10.250.77.85 (10.250.77.85) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/mkli/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=NG; L=Lagos; O=TekNeed; CN=*.tekneed.com
*  start date: Oct  8 00:00:00 2025 GMT
*  expire date: Oct  7 23:59:59 2026 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: 10.250.77.85
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: application/json; charset=utf-8
< Content-Length: 41
< Connection: keep-alive
< Date: Sun, 30 Aug 2026 19:17:06 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
* Connection #0 to host 10.250.77.85 left intact
{"message":"no match found","status":404}

The -k is important here because it tells curl:

“Don’t reject the connection just because the certificate doesn’t match the IP.”

That lets us continue past certificate validation and see what HTTP response Avi gives us.

And that is exactly why the next test with --resolve is even better: it lets us connect to the IP while still telling TLS/HTTP that the intended hostname is prod-tkg.fintechapi.tekneed

So, this is what it means.

9a. Direct connection to the VIP succeeded

* Trying 10.250.77.85...
* Connected to 10.250.77.85 (10.250.77.85) port 443 (#0)

This is very important.

You have directly established a TCP connection to:

10.250.77.85:443

So:

Network connectivity to the VIP on port 443 = working.

No proxy was involved.

9b. TLS handshake succeeded

You then see:

* TLSv1.3 (OUT), TLS handshake, Client hello
* TLSv1.3 (IN), TLS handshake, Server hello
* TLSv1.2 (IN), TLS handshake, Certificate

Your client and the server successfully performed the TLS handshake.

Eventually:

* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

So:

TLS connection = established.

9c. The certificate doesn’t match the IP — but `-k` tells curl to ignore that

The server presents:

CN=*.tekneed.com

while you connected to:

10.250.77.85

Normally, curl would complain:

SSL: no alternative certificate subject name matches target host name

But you used:

-k

which means:

Don’t perform normal certificate validation.

That’s why you get:

* SSL certificate verify ok.

Be careful with that message: because of -k, this does not mean the certificate matches the IP. It means curl is proceeding without enforcing the normal certificate verification.

9d. Curl sends the HTTP request

Now this is important:

> GET / HTTP/1.1
> Host: 10.250.77.85

Curl is saying to the server:

Give me /, and my Host is 10.250.77.85.

So the request is:

HTTPS
   |
   | Host: 10.250.77.85
   |
   v
Avi VIP
10.250.77.85

9e. You get an HTTP 404

Avi/application responds:

< HTTP/1.1 404 Not Found

and:

{"message":"no match found","status":404}

This is actually useful.

It means you didn’t get:

Connection refused

or:

Connection timed out

or:

TLS handshake failed

You got a legitimate HTTP response.

So the request travelled through:

Your machine
     |
     | TCP 443
     ↓
10.250.77.85
     |
     | TLS
     ↓
HTTPS listener
     |
     | HTTP GET /
     ↓
Avi / application routing
     |
     ↓
404 response

But there is one important limitation

You connected using:

https://10.250.77.85

rather than:

https://prod-tkg.fintechapi.tekneed

Therefore the HTTP Host header is:

Host: 10.250.77.85

That’s not necessarily the hostname configured in the Ingress/Virtual Service.

Your actual Ingress configuration expects a hostname such as:

prod-tkg.fintechapi.tekneed

So this test proves:

The VIP is directly reachable on HTTPS and is responding.

But it does not yet fully prove that the hostname-based Ingress routing works, because we’re sending the wrong Host header.

That’s why the next test is the most meaningful one:

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

That gives us the best combination:

DNS          → bypassed
Proxy        → bypassed
Destination  → 10.250.77.85
Hostname     → prod-tkg.fintechapi.tekneed
Host header  → prod-tkg.fintechapi.tekneed
TLS hostname → prod-tkg.fintechapi.tekneed

That is the closest thing to testing the real production URL while temporarily bypassing the DNS problem.

What if the hosts (URL) is not resolvable?

[Tekneed-victor@tekneedpkscli01 ~]$ k get ing

NAME               CLASS    HOSTS                   ADDRESS        PORTS     AGE
fintechai-tekneed  avi-lb   prod-tkg.fintechapi.tekneed   10.250.77.85   80, 443   33d

[Tekneed-victor@tekneedpkscli01 ~]$ nslookup prod-tkg.fintechapi.tekneed

Server:         10.22.134.23
Address:        10.22.134.23#53

** server can't find prod-tkg.fintechapi.tekneed: NXDOMAIN

That was why we mentioned above that we can temporarily bypass the DNS problem.

But before we use the command,

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

Lets run some curl commands against the host (URL)

[Tekneed-victor@tekneedpkscli01 ~]$ curl https://prod-tkg.fintechapi.tekneed

curl: (56) Received HTTP code 503 from proxy after CONNECT

[Tekneed-victor@tekneedpkscli01 ~]$ curl -v https://prod-tkg.fintechapi.tekneed

* Rebuilt URL to: https://prod-tkg.fintechapi.tekneed/
* Uses proxy env variable http_proxy == 'http://10.212.1.11:8080'
*   Trying 10.212.1.11...
* TCP_NODELAY set
* Connected to 10.212.1.11 (10.212.1.11) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to prod-tkg.fintechapi.tekneed:443
> CONNECT prod-tkg.fintechapi.tekneed:443 HTTP/1.1
> Host: prod-tkg.fintechapi.tekneed:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 503 Service Unavailable
< Mime-Version: 1.0
< Date: Sun, 30 Aug 2026 20:46:52 WAT
< Via: 1.1 teknsw02.tekneed.com:80 (Cisco-GMA/17.2.5-214)
< Content-Type: text/html
< Connection: keep-alive
< Proxy-Connection: keep-alive
< Content-Length: 2319
<
* Received HTTP code 503 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 503 from proxy after CONNECT

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -v https://prod-tkg.fintechapi.tekneed

* Rebuilt URL to: https://prod-tkg.fintechapi.tekneed/
* Could not resolve host: prod-tkg.fintechapi.tekneed
* Closing connection 0
curl: (6) Could not resolve host: prod-tkg.fintechapi.tekneed

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -vk https://prod-tkg.fintechapi.tekneed

* Rebuilt URL to: https://prod-tkg.fintechapi.tekneed/
* Could not resolve host: prod-tkg.fintechapi.tekneed
* Closing connection 0
curl: (6) Could not resolve host: prod-tkg.fintechapi.tekneed

We tested:

curl https://prod-tkg.fintechapi.tekneed
curl -v https://prod-tkg.fintechapi.tekneed
curl --noproxy '*' -v https://prod-tkg.fintechapi.tekneed
curl --noproxy '*' -vk https://prod-tkg.fintechapi.tekneed

1. First test

curl https://prod-tkg.fintechapi.tekneed

We got:

curl: (56) Received HTTP code 503 from proxy after CONNECT

Don’t focus on curl error 56 yet.

The important part is:

503 from proxy after CONNECT

That tells us the corporate proxy is involved and returned the 503.

The request path is roughly:

Our machine
     |
     v
Corporate Proxy
10.212.1.11:8080
     |
     X
    503

So this test doesn’t tell us whether the application itself is returning 503.

2. Now we used `-v`

We ran:

curl -v https://prod-tkg.fintechapi.tekneed

-v means:

Show me the communication details.

And now we can see exactly what happened.

Step 2.1 – Curl rebuilds the URL

* Rebuilt URL to: https://prod-tkg.fintechapi.tekneed/

Nothing unusual.

Curl understands:

HTTPS
Host = prod-tkg.fintechapi.tekneed
Port = 443
Path = /

2.2. Curl detects the HTTPS proxy

This is the important line:

* Uses proxy env variable http_proxy == 'http://10.212.1.11:8080'

Our machine has a proxy configured:

10.212.1.11:8080

So curl does not initially connect directly to:

prod-tkg.fintechapi.tekneed:443

Instead:

Our machine
      |
      | TCP
      v
10.212.1.11:8080
Corporate Proxy

2.3. Our machine successfully connects to the proxy

You see:

* Trying 10.212.1.11...
* Connected to 10.212.1.11 (10.212.1.11) port 8080

This proves:

Our machine can reach the corporate proxy on port 8080.

It does not prove that the application is reachable.

2.4. Curl asks the proxy to create an HTTPS tunnel

This is the key part:

* Establish HTTP proxy tunnel to prod-tkg.fintechapi.tekneed:443

Curl sends:

> CONNECT prod-tkg.fintechapi.tekneed:443 HTTP/1.1
> Host: prod-tkg.fintechapi.tekneed:443

Think of CONNECT as:

“Proxy, please open a TCP tunnel to this HTTPS destination on port 443.”

The desired path would be:

Your machine
     |
     v
Corporate Proxy
     |
     | CONNECT
     v
prod-tkg.fintechapi.tekneed:443

2.4. But the proxy returns 503

Instead of:

HTTP/1.1 200 Connection established

We got:

< HTTP/1.1 503 Service Unavailable

And curl explicitly tells us:

* Received HTTP code 503 from proxy after CONNECT

This is the crucial point.

The HTTPS tunnel was never established.

The sequence stopped here:

Your machine
     |
     | CONNECT hostname:443
     v
Corporate Proxy
     |
     X
     503

You never reached:

TLS handshake

You never received:

Server certificate

You never sent:

GET /

Therefore, this test tells us nothing about whether the application itself is healthy.

The 503 is from the proxy.

3. Now we removed the proxy

We then ran:

curl --noproxy '*' -v https://prod-tkg.fintechapi.tekneed

This changes the path.

Instead of:

Your machine
     |
     v
Proxy
     |
     v
Application

we are asking for:

Our machine
     |
     | DIRECT
     v
prod-tkg.fintechapi.tekneed:443

And now something different happens.

We got:

* Could not resolve host: prod-tkg.fintechapi.tekneed

This is a DNS problem.

3.1. Why did removing the proxy produce a DNS error?

Because now curl needs your machine itself to resolve:

prod-tkg.fintechapi.tekneed

into an IP address.

Conceptually:

prod-tkg.fintechapi.tekneed
              |
              | DNS lookup
              v
          ??? IP address

But DNS doesn’t provide one.

Therefore curl stops immediately.

It never gets to:

TCP
TLS
HTTP
Application

The process is:

DNS
 ↓
FAILED

That’s why:

curl: (6) Could not resolve host

4. Adding `-k` doesn’t change anything

We then tried:

curl --noproxy '*' -vk https://prod-tkg.fintechapi.tekneed

You might wonder:

“Shouldn’t -k help?”

No.

-k means:

Ignore TLS certificate verification problems.

But we are not even reaching TLS yet.

The request is failing at:

DNS

The sequence is:

DNS
 ↓
❌ FAILED
 ↓
TLS never starts

So -k has nothing to do at this stage.

Put all three tests side by side

This is the easiest way to understand what happened.

Test 1 – Normal curl

curl https://prod-tkg.fintechapi.tekneed

Path:

Our machine
     ↓
Corporate Proxy
     ↓
503

Failure point: Proxy

Test 2 – Verbose normal curl

curl -v https://prod-tkg.fintechapi.tekneed

We can see:

Our machine
     ↓
10.212.1.11:8080
     ↓
CONNECT hostname:443
     ↓
503

Failure point: Proxy

Test 3 – Bypass proxy

curl --noproxy '*' -v https://prod-tkg.fintechapi.tekneed

Path:

Our machine
     ↓
DNS lookup
     ↓
❌ Could not resolve host

Failure point: DNS

Test 4 – Bypass proxy + ignore certificate verification

curl --noproxy '*' -vk https://prod-tkg.fintechapi.tekneed

Same thing:

Our machine
     ↓
DNS lookup
     ↓
❌ Could not resolve host

-k never gets a chance to do anything.

The Better Test: `curl --resolve`

This is one of the most useful troubleshooting commands when DNS is broken.

Use:

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

What does this do?

It tells curl:

For this particular test, pretend that prod-tkg.fintechapi.tekneed resolves to 10.250.77.85.

You are effectively bypassing the normal DNS lookup.

The important thing is that you are not replacing the hostname with the IP.

You are still using:

prod-tkg.fintechapi.tekneed

as the hostname.

You’re simply telling curl which IP to use.

Why `--resolve` Is Better Than Curling the IP

Compare the two.

Test 1

curl https://10.250.77.85

You’re testing:

IP → Avi

The original hostname isn’t being used in the URL.

Test 2

curl --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
https://prod-tkg.fintechapi.tekneed/

You’re testing:

Hostname
    ↓
10.250.77.85
    ↓
Avi
    ↓
Virtual Service
    ↓
Pool
    ↓
Pods

This is much closer to what a real client would do.

What Does `--noproxy '*'` Do?

In many enterprise environments, servers have proxy variables configured.

For example:

http_proxy=http://10.250.178.150:8080

Then when you run:

curl https://prod-tkg.fintechapi.tekneed

curl may do:

Your server
    ↓
Corporate Proxy
    ↓
Destination

instead of:

Your server
    ↓
Avi VIP

This can produce errors such as:

HTTP/1.1 503
Received HTTP code 503 from proxy after CONNECT

or:

403 Proxy Unacknowledged

These errors may therefore be generated by the proxy, not by Avi and not by your application.

Using:

--noproxy '*'

tells curl:

Do not use the configured proxy. Connect directly.

So the troubleshooting command becomes:

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

Now lets run the command.

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -vk   --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85   https://prod-tkg.fintechapi.tekneed/

* Added prod-tkg.fintechapi.tekneed:443:10.250.77.85 to DNS cache
* Hostname prod-tkg.fintechapi.tekneed was found in DNS cache
*   Trying 10.250.77.85...
* TCP_NODELAY set
* Connected to prod-tkg.fintechapi.tekneed (10.250.77.85) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/mkli/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=NG; L=Lagos; O=TekNeed; CN=*.tekneed.com
*  start date: Oct  8 00:00:00 2025 GMT
*  expire date: Oct  7 23:59:59 2026 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: prod-tkg.fintechapi.tekneed
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: application/json; charset=utf-8
< Content-Length: 41
< Connection: keep-alive
< Date: Sun, 30 Aug 2026 21:27:39 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
* Connection #0 to host prod-tkg.fintechapi.tekneed left intact

[Tekneed-victor@tekneedpkscli01 ~]$ curl --noproxy '*' -vk http://prod-tkg.fintechapi.tekneed

* Rebuilt URL to: http://prod-tkg.fintechapi.tekneed
* Could not resolve host: prod-tkg.fintechapi.tekneed
* Closing connection 0
curl: (6) Could not resolve host: prod-tkg.fintechapi.tekneed

[Tekneed-victor@tekneedpkscli01 ~]$ curl prod-tkg.fintechapi.tekneed

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Notification: Policy Acknowledgement</title>
<style type="text/css">
body {
  font-family: Arial, Helvetica, sans-serif;
  font-size: 14px;
  color:#333333;
  background-color: #ffffff;
}
h1 {
  font-size: 18px;
  font-weight: bold;
  text-decoration: none;
  padding-top: 0px;
  color: #2970A6;
}
a:link {
    color: #2970A6;
  text-decoration: none;
}
a:hover {
    color: #2970A6;
  text-decoration: underline;
}
p.buttonlink {
  margin-bottom: 24px;
}
.copyright {
  font-size: 12px;
  color: #666666;
  margin: 5px 5px 0px 30px;

}
.details {
  font-size: 14px;
  color: #969696;
  border: none;
  padding: 20px 20px 20px 20px;
  margin: 0px 10px 10px 35px;
}

.shadow {
  border: 3px solid #9f9f9f;
  padding: 10px 25px 10px 25px;
  margin: 10px 35px 0px 30px;
  background-color: #ffffff;
  width: 600px;

  -moz-box-shadow: 3px 3px 3px #cccccc;
  -webkit-box-shadow: 3px 3px 3px #cccccc;
  box-shadow: 3px 3px 3px #cccccc;
  /* For IE 8 */
  -ms-filter: "progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc')";
  /* For IE 5.5 - 7 */
  filter: progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc');
}
.logo {
  border: none;
  margin: 5px 5px 0px 30px;
}
</style>
</head>

<body>
<div class="logo"></div><p>&nbsp;</p>
<div class="shadow">

<h1>URL Category Warning Acknowledgement</h1>


<p>
Please acknowledge the following statements before proceeding to the requested URL.
</p>

<p>
You are trying to visit a web page that falls under the URL Category Finance . By clicking the link below, you acknowledge that you have read and agree with the organization's policies that govern the usage of the Internet for this type of content. Data about your browsing behavior may be monitored and recorded. You will be periodically asked to acknowledge this statement for continued access to this kind of web page.
</p>



<p>
<a href="http://teknsw02.tekneed.com/A1:3600:14400:000000006a94a14c/60fcd22822d90b0510b8b93a5a46ac60e2c9b5e69d6bb1283bafb961dc311583/1786157335/http://prod-tkg.fintechapi.tekneed">Click here to accept this statement and access the Internet.</a>
</p>
</div>

<div class="details"><p>
Date: Sun, 30 Aug 2026 22:31:56 WAT<br />
Username: <br />
Source IP: 10.1.163.7<br />
</p></div>

</body>
</html>

Let’s go through the three tests separately.

1. This command

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

This is your most useful test.

Step 1 — `--noproxy '*'`

--noproxy '*'

This tells curl:

Do not use the corporate HTTP/HTTPS proxy. Connect directly.

So you’re removing the proxy from the troubleshooting path.

The connection becomes:

Your server
    |
    | direct connection
    v
10.250.77.85
    |
    v
Avi Virtual Service

Instead of:

Your server
    |
    v
Corporate Proxy
10.212.1.11:8080
    |
    v
10.250.77.85

That’s important because your previous tests were being affected by the proxy.

2. What does `--resolve` do?

This is the key part:

--resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85

It tells curl:

“For this particular request, don’t ask DNS where prod-tkg.fintechapi.tekneed is. Use 10.250.77.85.”

So you’re effectively creating a temporary DNS mapping inside curl:

prod-tkg.fintechapi.tekneed
             |
             | --resolve
             v
       10.250.77.85

It does not modify your DNS server.

It does not create a permanent DNS record.

It only applies to that curl command.

3. Look at this line

* Added prod-tkg.fintechapi.tekneed:443:10.250.77.85 to DNS cache

This means curl accepted your --resolve instruction.

Then:

* Hostname prod-tkg.fintechapi.tekneed was found in DNS cache

Don’t misunderstand this.

It did not suddenly find the hostname from your corporate DNS.

It found the mapping that you manually supplied with --resolve.

4. Then this happened

* Trying 10.250.77.85...

Curl is now connecting to:

10.250.77.85

And:

* Connected to prod-tkg.fintechapi.tekneed (10.250.77.85) port 443 (#0)

This is very important.

It proves:

Your server was able to establish a TCP connection to 10.250.77.85 on port 443.

So connectivity to the Avi VIP is working.

5. Then TLS started

You see:

* TLSv1.3 (OUT), TLS handshake, Client hello

and then:

* TLSv1.3 (IN), TLS handshake, Server hello

followed by the certificate:

* TLSv1.2 (IN), TLS handshake, Certificate

So Avi is responding to the HTTPS request and presenting a certificate.

You then see:

* Server certificate:
* subject: ... CN=*.tekneed.com

And importantly:

* SSL certificate verify ok.

So the TLS handshake succeeded.

Therefore:

TCP connectivity       SUCCESS
       ↓
Avi reachable          SUCCESS
       ↓
TLS handshake          SUCCESS
       ↓
Certificate validation SUCCESS

6. Then comes the REALLY important part

Look at:

> GET / HTTP/1.1
> Host: prod-tkg.fintechapi.tekneed

This is excellent for troubleshooting.

You are connecting to:

10.250.77.85

but you’re telling Avi:

Host: prod-tkg.fintechapi.tekneed

That’s exactly what you want.

Why?

Because Avi uses the hostname to determine which Virtual Service / application configuration should handle the request.

Conceptually:

                    10.250.77.85
                         |
                         v
                  +-------------+
                  |     AVI     |
                  |             |
                  | Host header |
                  |     ↓       |
                  | prod-tkg... |
                  +------+------+
                         |
                         v
                       Pool
                         |
                         v
                       Pods

7. Then Avi/application returned this

< HTTP/1.1 404 Not Found

And:

{"message":"no match found","status":404}

This is where we need to be precise.

Does this mean the application is reachable?

Yes, at least to the HTTP application layer.

The request was not stopped at DNS.

It wasn’t stopped by the proxy.

It wasn’t stopped by TCP connectivity.

It wasn’t stopped by TLS.

It reached something behind the VIP that understood the HTTP request and returned a valid HTTP response.

Your path was:

curl
 |
 | HTTPS
 v
10.250.77.85
 |
 v
Avi
 |
 v
Virtual Service
 |
 v
Pool
 |
 v
Application
 |
 v
HTTP response: 404

However…

Does `404` mean the application is healthy?

Not necessarily.

It means:

“The server handling this request couldn’t find a matching resource/route for /.”

Your application may simply not have a / endpoint.

For example, the actual application might expect:

/api/v1/customer

or:

/microgateway/...

rather than:

So / returning 404 can be perfectly normal.

8. Now compare this with your HTTP test

We also ran:

curl --noproxy '*' -vk http://prod-tkg.fintechapi.tekneed

and got:

Could not resolve host

That’s because --noproxy removed the proxy, but you didn’t provide --resolve.

So curl tried to perform normal DNS resolution:

prod-tkg.fintechapi.tekneed
             |
             v
          DNS
             |
             X
         NXDOMAIN

Therefore it never reached Avi.

That’s completely different from the HTTPS test.

9. And what about this?

curl prod-tkg.fintechapi.tekneed

We got the:

URL Category Warning Acknowledgement

page.

This is NOT our application.

This is the corporate proxy/security system.

The curl request went something like:

curl
 |
 v
Corporate Proxy
10.212.1.11:8080
 |
 v
Cisco security/web filtering
 |
 v
Policy Acknowledgement page

The HTML itself gives it away:

URL Category Warning Acknowledgement

and:

Via: ... Cisco-GMA ...

So don’t use that result to conclude that the application is working.

The whole situation in one picture

We currently have several different paths.

Test A – URL through corporate proxy

curl https://prod-tkg.fintechapi.tekneed

Result:

503 from proxy

Meaning:

Your server
   ↓
Corporate Proxy
   ↓
Proxy cannot/doesn't want to establish connection
   X

Your application has not necessarily been reached.

Test B – URL without proxy

curl --noproxy '*' https://prod-tkg.fintechapi.tekneed

Result:

Could not resolve host

Meaning:

Your server
   ↓
DNS
   X
NXDOMAIN

The request doesn’t get anywhere near Avi.

Test C – IP directly

curl --noproxy '*' -vk https://10.250.77.85

Result:

Connected
TLS handshake
Certificate
404

But the Host header is:

Host: 10.250.77.85

So this is not the ideal application test, because you’re identifying yourself to Avi as the IP rather than as the application hostname.

Test D – `--resolve`

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

This is the best test we have performed so far.

It gives us:

DNS bypassed                 ✓
Corporate proxy bypassed     ✓
TCP connection to VIP        ✓
Avi reachable                ✓
TCP 443 reachable            ✓
TLS handshake                ✓
Certificate validation       ✓
Correct Host header          ✓
HTTP request reached server  ✓
HTTP response received       ✓

And the response was:

404 Not Found
{"message":"no match found","status":404}

So yes, this demonstrates successful end-to-end reachability to the HTTP layer through the VIP, but it does not by itself prove that the particular application API/function you need is healthy.

The most important troubleshooting concept

When troubleshooting this type of setup, think in layers:

                 prod-tkg.fintechapi.tekneed
                           |
                           v
                    [ DNS resolution ]
                           |
                           v
                     10.250.77.85
                           |
                           v
                    [ TCP port 443 ]
                           |
                           v
                         [ AVI ]
                           |
                           v
                    [ TLS / HTTPS ]
                           |
                           v
                  [ Host / URL routing ]
                           |
                           v
                        [ Pool ]
                           |
                           v
                         [ Pods ]
                           |
                           v
                    [ Application ]

Your --resolve test has successfully taken you past all of these up to receiving an HTTP response.

The next question is therefore no longer:

“Can I reach the VIP?”

We pretty much demonstrated that.

The next question is:

“Why does the application return 404 for /, and what is the correct application/API path I should test?”

And separately, the DNS problem remains:

Why does normal DNS return NXDOMAIN for prod-tkg.fintechapi.tekneed?

Those are now two separate troubleshooting issues, and keeping them separate will make this much easier.

Understanding the Successful Test

Suppose you run:

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/

and see:

* Added prod-tkg.fintechapi.tekneed:443:10.250.77.85 to DNS cache
* Trying 10.250.77.85...
* Connected to prod-tkg.fintechapi.tekneed (10.250.77.85) port 443

This tells us:

Hostname
   ↓
forced to VIP
   ↓
TCP connection successful

Then you may see:

* SSL connection using TLSv1.2

and:

* Server certificate:
* subject: CN=*.fintechapi.tekneed

That means the TLS handshake succeeded.

Then curl sends:

GET / HTTP/1.1
Host: prod-tkg.fintechapi.tekneed

Notice the important part:

Host: prod-tkg.fintechapi.tekneed

This means we are testing the actual hostname-based application routing.

What Does a 404 Mean Here?

Suppose the response is:

HTTP/1.1 404 Not Found

{"message":"no match found","status":404}

This is very different from:

Could not resolve host

A 404 means the request got much further.

You have potentially established:

DNS bypass
    ↓
Network connectivity
    ↓
Avi VIP
    ↓
TLS
    ↓
Virtual Service
    ↓
Application routing

The 404 may simply mean that:

GET /

is not a valid application endpoint.

For example, the application might require:

/api

or:

/v1/customer

or some other path.

Therefore, a 404 from the application is not automatically evidence that the Ingress is broken.

In fact, it can be useful evidence that the request reached the application stack.

Understanding the Avi Virtual Service

Now let’s look at what happens inside Avi.

A Kubernetes Ingress can cause AKO to create an Avi Virtual Service.

Think of a Virtual Service as the frontend listener for the application.

For example:

Virtual Service:

Name:
apimrois-01-prod-tkg.fintechapi.tekneed-L7-dedicated

VIP:
10.250.77.85

Hostname:
prod-tkg.fintechapi.tekneed

Ports:
80
443

Conceptually:

                     AVI
                      |
              Virtual Service
                      |
              VIP 10.250.77.85
                      |
          -----------------------
          |                     |
        TCP 80                TCP 443
                              TLS

Why Does Avi Show Port 80 and 443?

This can initially be confusing because you might look at the Ingress YAML and not see:

port: 80
port: 443

The Ingress backend port is:

port:
  number: 30084

That is a Kubernetes Service port.

It is not the same thing as the external frontend port on the Avi Virtual Service.

The architecture has different ports at different layers:

Client
   |
   | HTTPS 443
   ↓
Avi Virtual Service
   |
   | backend connection
   ↓
Kubernetes Service 30084
   |
   | targetPort 8000
   ↓
Pod 8000

So:

is the external frontend port.

While:

is the Kubernetes Service port.

And:

is the Pod/application port.

These are three different things.

What About TLS?

The Ingress contains:

tls:
- hosts:
  - prod-tkg.fintechapi.tekneed
  secretName: wildcard-fintechapi-2026-cert

This tells AKO that TLS should be configured for the hostname.

Avi can therefore configure the Virtual Service with HTTPS/TLS on:

You may consequently see in Avi:

Service Ports:

80
443 (SSL)

The:

(SSL)

means Avi is treating that listener as an SSL/TLS service.

So the relationship is:

Ingress TLS configuration
          ↓
          AKO
          ↓
Avi Virtual Service
          ↓
443 SSL/TLS

Understanding the Avi Pool

This is another concept that can be confusing when you first start using Avi.

Inside the Virtual Service, you may see something like:

Pool:
apimrois-01--apimrois-01-prod-tkg.fintechapi.tekneed_-fintechai-tekneed-L7-dedicated

What is a Pool?

A Pool is a group of backend servers that Avi can send traffic to.

In a traditional application environment, those servers might be:

Web Server 1
Web Server 2
Web Server 3

But in Kubernetes, those backend servers can be Pod IP addresses.

For example:

Pool

Port     Server
8000     10.250.24.129
8000     10.250.33.117

This means Avi has two backend members.

                  Virtual Service
                         |
                         ↓
                       Pool
                    /         \
                   /           \
                  ↓             ↓
       10.250.24.129       10.250.33.117
             :8000               :8000
                  \             /
                   \           /
                    Application

Why Are the Pool IPs Pod IPs?

Because the Kubernetes application ultimately runs inside Pods.

Suppose we have:

Pod 1
10.250.24.129:8000

Pod 2
10.250.33.117:8000

Avi can use those as backend servers.

Therefore:

Avi Pool
   |
   +--- 10.250.24.129:8000
   |
   +--- 10.250.33.117:8000

This means traffic arriving at the Avi Virtual Service can ultimately be forwarded to one of those Pods.

The Complete Traffic Flow

Now we can put everything together.

A user enters:

https://prod-tkg.fintechapi.tekneed

The complete flow is:

1. DNS
   |
   | prod-tkg.fintechapi.tekneed
   ↓
   10.250.77.85

2. Avi VIP
   |
   ↓

3. Avi Virtual Service
   |
   | HTTPS 443
   ↓

4. Avi Pool
   |
   +------ 10.250.24.129:8000
   |
   +------ 10.250.33.117:8000
   |
   ↓

5. Application Pod

But where does Kubernetes Service 30084 fit?

Conceptually, the Kubernetes configuration says that:

Service 30084
       ↓
targetPort 8000
       ↓
Pods

Service 30084
       ↓
targetPort 8000
       ↓
Pods

The exact backend implementation can depend on how AKO is configured and how it programs Avi, but from the Kubernetes perspective the Ingress backend is:

Ingress
   ↓
Service fintechai-tekneed:30084
   ↓
Pod/application port 8000

Understanding the Avi Pool Health

When you opened the Pool in Avi, you might see something like:

Port     Server IP          Health
8000     10.250.24.129      100
8000     10.250.33.117      100

A health value of:

means that Avi considers that backend member healthy according to its configured health monitoring.

This is important because Avi does not blindly send traffic to every backend.

It can monitor the backend servers and determine whether they are available.

For example:

Pool

Pod A     Healthy
Pod B     Healthy
Pod C     Down

Avi can avoid sending traffic to Pod C.

What Does RPS Mean?

When looking at Avi analytics, you may see:

RPS

RPS means:

Requests Per Second

It tells you approximately how many requests the Virtual Service or backend is handling every second.

For example:

RPS: 85.5/sec

means approximately:

85.5 requests every second

If you see:

RPS: 0.0/sec

there may currently be little or no traffic.

RPS is different from:

Open Connections

which refers to connections currently open.

Understanding Avi Virtual Service Health

You may also see a Virtual Service health score such as:

while other Virtual Services have:

This does not necessarily mean:

“The application is down.”

Avi’s overall health score can be calculated from multiple factors.

For example, you may see factors such as:

Performance Score
Resources Score
Anomaly Score
Security Threat Level

Avi may also show factors such as:

Apdex
Service Engine Anomaly Scores
Anomalous L7 Metrics

Therefore, a Virtual Service health score of 38 should be investigated rather than immediately interpreted as:

Application = Down

For example, the backend pool may still show:

Pod A → Health 100
Pod B → Health 100

while Avi’s overall Virtual Service health score is lower because of performance, anomaly, or other metrics.

This is why it is useful to check:

Virtual Service
      ↓
Health
      ↓
Analytics
      ↓
Pool
      ↓
Pool members

rather than relying on only one number.

Understanding the Difference Between Virtual Service Health and Pool Health

This is an important distinction.

You could have:

Virtual Service Health: 38

while:

Pool Member 1: 100
Pool Member 2: 100

There is no contradiction.

The pool health answers something like:

Are my backend servers responding according to the configured health monitor?

The Virtual Service health score considers a broader set of metrics about the service.

Therefore, don’t automatically conclude:

VS health = 38

means:

Pods are unhealthy.

You need to investigate the specific factors contributing to the VS score.

What Does `L7-dedicated` Mean?

You may see a Virtual Service named something like:

apimrois-01--prod-tkg.fintechapi.tekneed-L7-dedicated

The important part is:

L7

This indicates Layer 7 processing.

The dedicated portion generally relates to the Virtual Service/Service Engine placement or allocation configuration used by Avi.

The exact meaning depends on the Avi configuration in the environment, so it should not simply be interpreted as “a dedicated physical load balancer.”

The key takeaway is:

L7 = Layer 7 application-aware traffic handling

Understanding the AKO Annotation

When you inspect the Ingress, you may see:

annotations:
  ako.vmware.com/host-fqdn-vs-uuid-map: '{"prod-tkg.fintechapi.tekneed":"virtualservice-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}'

This is very useful.

It establishes a relationship between:

Hostname

and:

Avi Virtual Service

Conceptually:

prod-tkg.fintechapi.tekneed
             |
             ↓
Avi Virtual Service UUID
             |
             ↓
Virtual Service
             |
             ↓
VIP

This is one way AKO tracks the Avi object associated with the Kubernetes Ingress.

Why Might You Temporarily Not Find the Virtual Service in Avi?

Suppose Kubernetes reports:

virtualservice-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

but you search Avi and don’t immediately find it.

There are several things worth checking before concluding that the configuration is broken.

For example:

correct Avi Controller
correct tenant
exact UUID
current Virtual Service name
whether AKO has reconciled the configuration
whether the Virtual Service was renamed or recreated
whether you are looking at the correct Avi object/interface

The important point is that the Kubernetes annotation gives you a valuable identifier for tracing the relationship.

The Complete Architecture

Let’s put everything together.

                    USER / CLIENT
                          |
                          |
                          | HTTPS
                          | 443
                          ↓
                 prod-tkg.fintechapi.tekneed
                          |
                          |
                          | DNS
                          ↓
                    10.250.77.85
                          |
                          |
                          ↓
              ┌─────────────────────┐
              │   AVI VIP / VS      │
              │                     │
              │ Port 80             │
              │ Port 443 SSL        │
              └──────────┬──────────┘
                         |
                         |
                         ↓
                  AVI L7 ROUTING
                         |
                         ↓
                    AVI POOL
                  /            \
                 /              \
                ↓                ↓
       10.250.24.129      10.250.33.117
            :8000               :8000
                \                /
                 \              /
                  ↓            ↓
                    PODS
                       |
                       |
                       ↓
                 APPLICATION

And on the Kubernetes side:

Ingress
   |
   | host:
   | prod-tkg.fintechapi.tekneed
   |
   | backend:
   | fintechai-tekneed:30084
   |
   ↓
Service
10.250.241.202:30084
   |
   | targetPort 8000
   ↓
Pods
10.250.24.129:8000
10.250.33.117:8000

The Most Important Concepts to Remember

If you are new to Kubernetes Ingress + Avi, remember these relationships.

Ingress

Defines:

Hostname
Path
Backend Service
TLS
Ingress Class

Service

Provides a stable Kubernetes endpoint for Pods.

For example:

Service port 30084
       ↓
targetPort 8000
       ↓
Pod

Pod

Runs the actual application.

Example:

10.250.24.129:8000
10.250.33.117:8000

AKO

Acts as the integration between Kubernetes and Avi.

Conceptually:

Kubernetes
    ↓
   AKO
    ↓
   Avi

Avi VIP

The frontend IP.

Example:

10.250.77.85

Avi Virtual Service

The frontend service listening on the VIP.

For example:

VIP: 10.250.77.85
Ports: 80, 443
Hostname: prod-tkg.fintechapi.tekneed

Avi Pool

The backend group that receives traffic.

For example:

10.250.24.129:8000
10.250.33.117:8000

DNS

Maps the user-friendly hostname to the VIP.

prod-tkg.fintechapi.tekneed
             ↓
       10.250.77.85

Final Picture

The easiest way to remember the entire architecture is this:

                         DNS
                          |
                          |
prod-tkg.fintechapi.tekneed
                          |
                          ↓
                   10.250.77.85
                     AVI VIP
                          |
                          ↓
                AVI Virtual Service
                     L7 / HTTPS
                     Port 443
                          |
                          ↓
                     AVI Pool
                    /         \
                   /           \
                  ↓             ↓
        Pod 10.250.24.129   Pod 10.250.33.117
               :8000              :8000
                  \                 /
                   \               /
                    \             /
                     APPLICATION

While Kubernetes itself looks like:

Ingress
   |
   | Host: prod-tkg.fintechapi.tekneed
   |
   | Backend:
   | fintechai-tekneed:30084
   ↓
Service
   |
   | 30084 → 8000
   ↓
Pods
   |
   +--- 10.250.24.129:8000
   |
   +--- 10.250.33.117:8000

And the complete request path is:

User
 ↓
DNS
 ↓
Avi VIP
 ↓
Avi Virtual Service
 ↓
L7 routing
 ↓
Avi Pool
 ↓
Pod
 ↓
Application

Once you understand that flow, troubleshooting becomes much easier because you can identify which layer failed instead of treating “the application is not reachable” as one single problem.

Avoid the Corporate Proxy

Check whether your environment has a proxy:

env | grep -i proxy

You may see:

http_proxy=...
https_proxy=...

If necessary, bypass it:

curl --noproxy '*' ...

Check the EndpointSlice

Modern Kubernetes versions prefer EndpointSlices:

kubectl get endpointslice \
  -n fintech-production

You can also inspect the Service’s endpoints.

This helps answer:

Which Pods are currently behind this Service?

Test a Known Application Endpoint

If / returns:

don’t immediately conclude that the application is down.

Find out what endpoint the application actually exposes.

For example, if the application exposes:

/api/health

test:

curl --noproxy '*' -vk \
  --resolve prod-tkg.fintechapi.tekneed:443:10.250.77.85 \
  https://prod-tkg.fintechapi.tekneed/api/health

A known application endpoint provides a much better application-level test than simply requesting /.

Understanding the Subject Matter

What is Kubernetes Ingress?

Is Ingress Layer 7?

The Kubernetes Service Behind the Ingress

Understanding port and targetPort

What About Port 4317?

What Happens Behind the Service?

Now Introduce VMware Avi

The VIP

Was the VIP Manually Defined in the Ingress YAML?

The Ingress Doesn’t Need to Contain the VIP

Does the VIP Change When Avi or AKO Restarts?

DNS Comes Before the User Reaches the VIP

What Happens If DNS Is Not Configured?

But What If the VIP Is Reachable?

curl -v

1. Curl understands the URL

2. Curl sees that an HTTP proxy is configured

3. Curl connects to the proxy

4. Curl sends the HTTP request to the proxy

5. The proxy returns HTTP 301

6. Now look at the Avi Vantage part

7. Finally, curl closes its connection to the proxy

So what actually happened?

What we have proven

What we have not cleanly proven

Now, since our test asked us to use https, let’s do that.

1. First command: curl https://10.250.77.85

2. Curl builds the URL

3. Again, the proxy is involved

4. Curl asks the proxy to create an HTTPS tunnel

This is important.

5. TLS negotiation starts

6. But we requested the IP address

7. So did it reach Avi?

8. This is why curl https://IP is not the same as using the hostname

9. And notice something else: this test used the proxy

9a. Direct connection to the VIP succeeded

9b. TLS handshake succeeded

9c. The certificate doesn’t match the IP — but -k tells curl to ignore that

9d. Curl sends the HTTP request

9e. You get an HTTP 404

But there is one important limitation

What if the hosts (URL) is not resolvable?

1. First test

2. Now we used -v

Step 2.1 – Curl rebuilds the URL

2.2. Curl detects the HTTPS proxy

2.3. Our machine successfully connects to the proxy

2.4. Curl asks the proxy to create an HTTPS tunnel

2.4. But the proxy returns 503

The HTTPS tunnel was never established.

3. Now we removed the proxy

3.1. Why did removing the proxy produce a DNS error?

4. Adding -k doesn’t change anything

Put all three tests side by side

Test 1 – Normal curl

Test 2 – Verbose normal curl

Test 3 – Bypass proxy

Test 4 – Bypass proxy + ignore certificate verification

The Better Test: curl --resolve

Why --resolve Is Better Than Curling the IP

What Does --noproxy '*' Do?

1. This command

Step 1 — --noproxy '*'

2. What does --resolve do?

3. Look at this line

4. Then this happened

5. Then TLS started

6. Then comes the REALLY important part

7. Then Avi/application returned this

Does this mean the application is reachable?

Does 404 mean the application is healthy?

8. Now compare this with your HTTP test

9. And what about this?

The whole situation in one picture

Test A – URL through corporate proxy

Test B – URL without proxy

Test C – IP directly

Test D – --resolve

Understanding `port` and `targetPort`

6. Now look at the `Avi Vantage` part

1. First command: `curl https://10.250.77.85`

8. This is why `curl https://IP` is not the same as using the hostname

9c. The certificate doesn’t match the IP — but `-k` tells curl to ignore that

2. Now we used `-v`

4. Adding `-k` doesn’t change anything

The Better Test: `curl --resolve`

Why `--resolve` Is Better Than Curling the IP

What Does `--noproxy '*'` Do?

Step 1 — `--noproxy '*'`

2. What does `--resolve` do?

Does `404` mean the application is healthy?

Test D – `--resolve`

What Does `L7-dedicated` Mean?