100-Days-Of-DevOps-Challenge-KodeKloud

Linux Network Services

Our monitoring tool has reported an issue in Stratos Datacenter. One of our app servers has an issue, as its Apache service is not reachable on port 3000 (which is the Apache port). The service itself could be down, the firewall could be at fault, or something else could be causing the issue.

• Use tools like telnet, netstat, etc. to find and fix the issue. Also make sure Apache is reachable from the jump host without compromising any security settings.

• Once fixed, you can test the same using command curl http://stapp01:3000 command from jump host.

Steps

1. Login into app server.

2. Check httpd/apache/nginx service status

    tony@stapp01 ~]$ sudo systemctl status httpd
    ● httpd.service - The Apache HTTP Server
    Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset
    : disabled)
    Active: failed (Result: exit-code) since Wed 2025-08-06 01:38:21 UT
    C; 13min ago
        Docs: man:httpd.service(8)
    Process: 491 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exit
    ed, status=1/FAILURE)
    Main PID: 491 (code=exited, status=1/FAILURE)
    Status: "Reading configuration..."
   
    Aug 06 01:38:21 stapp01.stratos.xfusioncorp.com httpd[491]: (98)Address already i
    n use: AH00072: make_sock: could not bind to address 0.0.0.0:3000
    Aug 06 01:38:21 stapp01.stratos.xfusioncorp.com httpd[491]: no listening sockets 
    available, shutting down
    top -
   

3. Lets check the network port status

    sudo netstat -tlnup
   

    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
    tcp        0      0 127.0.0.11:36025        0.0.0.0:*               LISTEN      -                   
    tcp        0      0 127.0.0.1:3000          0.0.0.0:*               LISTEN      430/sendmail: accep 
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      298/sshd            
    tcp6       0      0 :::22                   :::*                    LISTEN      298/sshd            
    udp        0      0 127.0.0.11:56145        0.0.0.0:*                           -                   
   

   It’s clearly visible that the ‘3000’ port is already being used by sendmail

4. So Either we need to change port 3000 on sendmail or we can run httpd on different port. Since target is to run apache on 3000, we have to change sendmail port.

5. Changing sendmail port

    cd /etc/mail
    cp sendmail.mc sendmail.mc.bak
    vi sendmail.mc
   

   Find the following line and change port with some other value (i,e; 1234):

    DAEMON_OPTIONS(`Port=3000,Addr=127.0.0.1, Name=MTA')dnl
   

    sudo systemctl restart sendmail
   

6. Now lets check port and servicec status

    sudo netstat -tlnup
    sudo systemctl status httpd sendmail
   

7. Test

   From app server:

    curl http://localhost:3000
   

   From jump host:

    curl http://stapp01:3000
   

8. Debugging and Decision
   • From netstat we can see port 3000 listening on all interfaces.
   • ifconfig we can see jump host and app server connected with route
   • if we use telnet we see its giving no route to host. So we should check the firewall.

9. Fixing firewall using iptables

    sudo iptables -L -n
   

    sudo iptables -L -n
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
   
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
   
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         
    # Warning: iptables-legacy tables present, use iptables-legacy to see them
   

   That FORWARD rule is blocking the connection.

   Run the following command:

    sudo iptables -I INPUT 4 -p tcp --dport 3000 -j ACCEPT
   

10. Finally it should work: curl http://stapp01:3000

Good to Know?

Network Troubleshooting Tools

• netstat: Display network connections and listening ports
• ss: Modern replacement for netstat (faster, more features)
• telnet: Test TCP connectivity to specific ports
• nmap: Network discovery and port scanning
• lsof: List open files and network connections

Port Conflict Resolution

• Identification: Use netstat -tlnup or ss -tlnup
• Process Discovery: lsof -i :port shows which process uses port
• Solutions: Change port, stop conflicting service, or reconfigure
• Prevention: Document port assignments, use port ranges

Firewall Troubleshooting

• iptables: Linux firewall rules management
• Rule Order: Rules processed top-to-bottom, first match wins
• Common Issues: REJECT rules blocking legitimate traffic
• Testing: Use telnet or nc to test connectivity

Service Management

• systemctl: Control systemd services
• Service Dependencies: Some services depend on others
• Configuration: Check service-specific config files
• Logs: Use journalctl -u service-name for troubleshooting

This site is open source. Improve this page.