Debugging Nginx

Nginx, unlike most web servers, handles high traffic and scales with minimal hardware. Due to its event-driven architecture, it handles a large amount of requests with ease. Nginx is predominantly used in environments where you need to distribute client requests across multiple servers.

On top of all this, Nginx is an ideal option for load balancing, caching, and URL rewriting. This gives administrators the right to permit or restrict access for incoming requests.

What are Nginx logs?

Nginx records data related to interactions between the Nginx web servers and clients, such as browsers, as logs. These logs typically include information like client connections, web server traffic, resource utilization, and more.

Nginx provides different types of logging levels based on various degrees of severity and verbosity required:

• debug: Detailed messages for debugging purposes.
• info: Important events, such as server start and stop.
• notice: Events worth noting but not necessarily errors. For example, server startup, configuration reloads, and worker process events are events that are logged at the notice level.
• warn: Potential error conditions that may require attention.
• error: Actual error conditions that have occurred.
• crit: Severe error conditions requiring immediate attention.
• alert: Conditions requiring immediate action.
• emerg: Emergency conditions that may render the system unusable.

Why debugging mode?

As the logging level increases, the details that a log contains about a certain incident will be rich in verbosity. So when you run Nginx in debug mode, it will contain extensive data about all the activities and help you pinpoint and troubleshoot problems more easily.

How do I configure Nginx debug mode?

Configuring debug logging with Nginx involves modifying the Nginx configuration files to increase the verbosity of the information. When you encounter a problem and want to gain deeper insights of your Nginx server operations to find a work around, switch to debug logging by:

1. Ensure Nginx supports debugging

Verify that your Nginx installation supports debugging. This is typically indicated by the presence of the '--with-debug' flag during compilation. If your version does not support debugging, you may need to install a debug version of Nginx, such as 'nginx-debug' on CentOS.

2. Modify Nginx configuration

• Edit the Nginx configuration file: Open your main Nginx configuration file, usually located at '/etc/nginx/nginx.conf', or a site-specific configuration file in '/etc/nginx/sites-available/' or '/etc/nginx/conf.d/'.
• Set the debug level: Add or modify the error_log directive within the http, server, or location block to include the debug level.

error_log /var/log/nginx/error.log debug;

This will log detailed debug information to the specified file.

• Location specific debugging: If you want to debug a specific part of your application, you can add an error_log directive inside a location block:

server {
# Other configurations...
error_log /var/log/nginx/example.com.error.log;
location /admin/ {
error_log /var/log/nginx/admin-error.log debug;
}
}

3. Reload Nginx configuration:

After making changes to the configuration files, you need to reload Nginx to apply them. You can do this using one of the following commands:

• Ubuntu/Debian:
sudo nginx -t && sudo service nginx reload
• CentOS/RHEL:
sudo nginx -t && sudo systemctl reload nginx

4. Review debug logs:

• Location of logs: Debug logs are written to the file specified in the error_log directive, typically '/var/log/nginx/error.log'.
• Analyzing logs: Use tools like tail, grep, or log analysis software to review the logs for specific patterns or errors.

5. Disable debug logging

• Edit the configuration file: Open the configuration file again.
sudo nano /etc/nginx/nginx.conf
• Change the log level: Modify the error_log directive to reduce the log level. For example, change it to info:
error_log /var/log/nginx/error.log info;

Alternatively, you can comment out the error_log directive or set it to '/dev/null' for minimal logging:

#error_log /var/log/nginx/error.log;
# or
error_log /dev/null crit;
• Reload Nginx configuration:

Apply the changes by reloading Nginx:

sudo nginx -t && sudo service nginx reload

or

sudo nginx -t && sudo systemctl reload nginx

Why debug logging is important?

Debug logs provide detailed insights into the server's performance by picking up the trails and helping administrators troubleshoot issues faster and enhance overall security.

Debug logs help:

• Provide detailed error messages that help identify and remediate issues, such as misconfiguration and anomalous behavior.
• Provide detailed information about how Nginx processes requests, which can be crucial for debugging rewrite rules, proxy configurations, or other advanced setups.
• Identify bottlenecks or slow operations to optimize server performance and improve response times.
• Ensure that Nginx configurations are working as intended.

Implement complete observability for debug-level Nginx logs with ManageEngine Log360

ManageEngine Log360 is a cutting-edge SIEM solution that collects, parses, archives, and analyzes logs from various sources, including NGINX among other web servers. Log360's Nginx extension allows you to fortify the web server by ingesting logs and delivering comprehensive insights into web traffic, sensitive files accessed, and how well your server cache is functioning.

Log360's out-of-the-box reports bring you insights you need to get a complete overview of what's happening in your server. These reports help you:

• Monitor HTTP status codes for success, redirection, client/server errors, and informational responses.
• Analyze upstream requests, redirections, and cache efficiency, ensuring optimal server performance and security.
• Identify slow URI paths, virtual servers, and top clients generating 4xx errors, providing insights into potential abuse.
• Track overall web server traffic, request rates, URI access, virtual server activity, and SSL performance (success/failure trends).

Log360's correlation engine correlate Nginx data with network devices and cloud logs, to identify anomalies, offering deep insights into user behavior. This helps in detecting complex incidents like Log4Shell JNDI Payload Injection Attempt and server privilege escalation.

Log360 also helps your organization be compliant with standards like PCI DSS, HIPAA, and the GDPR. These audit-ready reports are just a click away and can be exported easily as and when required.

What's next?