HSTS or Strict Transport Security is a standard defined in RFC 6797, by which a web server can declare to a client that it should only be accessed via HTTPS. The web server or crawler will then make all future requests over HTTPS. This will be the case even if following a link to an HTTP URL. From here the SEO Spider shows a Status Code of 307 and a Status of HSTS Policy and a Redirect Type of HSTS Policy.
This redirect is an internal representation in the SEO Spider and the browser. This differs from a 301 or a 302 as it isn’t sent by the web server as its turned around internally. When a webserver declares it should be contacted through HTTPS an expiry on this – this 307 response is ideal as it means temporary re-direct.
Protocol
The HSTS protocol is based on the server sending a single header. This is called a Strict-Transport-Security and is only sent via HTTPS as if sent via HTTP it is overlooked. The header requires 2 associated directives max-age and includeSubDomains.
Max-age is mandatory and lets the server know the number of seconds in which it can only be contacted by HTTPS. IncludeSubDomains is an optional field which if set, signals that HSTS Policy applies to any sub-domains.
Benefits
There are several benefits to using HTTP – > HTTPS Redirect. It reduces the communication over non-secure protocols, reduces load on the web server as well as improving the performance as a round trip is avoided when the HTTP link is encountered.
A site-wide HTTP->HTTPS redirect is still needed due to the Strict-Transport-Security header ignoring this unless it is sent over the HTTPS. If the first visit to your site is not via HTTPS, you still need that initial redirect to HTTPS to deliver the Strict-Transport-Security header. Considering this, you may not expect to see a 307 in the SEO Spider, but makes an HTTP request for the robots.txt file, receives a 301 to the HTTPS version of the site, then receives the Strict-Transport-Security header, so will then report 307 for the first URL crawled. If robots.txt is disabled checking the SEO spider will report a 301.
How to disable HSTS
This can be easily done by unticking: ‘Respect HSTS Policy’ configuration under ‘Configuration > Spider > Advanced’ in the SEO Spider.
The SEO Spider will ignore HSTS completely and report upon the underlying redirects and status codes.
Leave a Reply