Which HTTP headers are most important for SEO?
Most impactful are X-Robots-Tag (controls indexing at the HTTP level, like noindex), Cache-Control (affects crawl efficiency and CDN behavior), Link rel="canonical" (alternative to HTML canonical tag), Content-Type (must match content, especially for non-HTML files), and Vary (signals to caches what changes the response). Compression headers like Content-Encoding (gzip, br) reduce page weight. Audit all of these on at least one URL per template to ensure correct configuration before scaling assumptions.
What security headers should every site set?
Strict-Transport-Security (HSTS, forces HTTPS), Content-Security-Policy (prevents XSS and injection), X-Content-Type-Options: nosniff (prevents MIME sniffing), X-Frame-Options or CSP frame-ancestors (prevents clickjacking), and Referrer-Policy (controls referrer leakage). Permissions-Policy is newer and limits browser feature access. Use Mozilla Observatory or securityheaders.com to grade your setup. Most sites scoring below B have easy wins: HSTS and X-Content-Type-Options each take 5 minutes to add and prevent real attacks.
Why does my CDN serve different headers than my origin?
CDNs intentionally rewrite, add, or strip headers for performance and security. They commonly add Server, Via, X-Cache headers (showing cache status), strip Cookie headers from cached responses, and override Cache-Control to extend TTLs. To inspect origin headers directly, bypass the CDN with curl directly to the origin IP using the Host header. Mismatches between origin and CDN headers cause caching bugs, especially when the origin sends Vary headers the CDN ignores or honors incompletely.
How can I tell if my caching headers are working correctly?
Look for Cache-Control with explicit max-age values (e.g., max-age=31536000 for static assets, max-age=300 for HTML), ETag or Last-Modified for conditional requests, and CDN-specific headers like CF-Cache-Status: HIT for Cloudflare or Age greater than 0 for any HTTP cache. If responses repeatedly show MISS or no-cache, your caching is not working. Also check that no-cache is not accidentally set on static assets, a common WordPress misconfiguration that ruins page speed scores.
What is the X-Robots-Tag header used for?
X-Robots-Tag applies meta-robots directives at the HTTP level, useful for non-HTML files like PDFs, images, and dynamic responses where you cannot insert a meta tag. Common uses are noindex on PDFs you do not want in search, nofollow on certain endpoints, and noarchive to prevent cached copies. Set via web server config or application headers. The advantage is bulk control: a single rule in nginx can noindex thousands of URLs without modifying each file individually.
Why do some headers appear with different cases or as duplicates?
HTTP header names are case-insensitive per spec, so Content-Type and content-type are equivalent. Some servers normalize case (HTTP/2 lowercases everything), others preserve it. Duplicate headers can come from your application, CDN, and reverse proxy each adding their own copy, especially for X-Powered-By or Server headers. Audit and remove duplicates, prefer the more specific value, and standardize on one source per header. Duplicate Set-Cookie is normal (one per cookie), other duplicates are usually configuration bugs.