Until now, all of the proxies I have created centered on independent websites and Blogger sites. These are fairly straight forward to proxy using Apache’s mod_proxy_html package and there are enough (but I won’t say plenty of-) tutorials online to throw something together that works most of the time. With all of the low-hanging fruit completed I decided to attempt to index pages on Tumblr. I had never visited Tumblr before, and now I realize that it is probably the most active scene when it comes to new Pirate101 fan content. Just a heads up, this post contains fairly technical information that you do not need to worry about, but some people may find it interesting/helpful.
I tried to proxy piecesofchess first. I started with the same steps I used for all of the other sites (which I need to post sometime soon), but only the landing page would successfully display under the p101 domain. Every link would lead to an HTTPS version of the site despite displaying a regular HTTP url preview. Right now this network only functions on HTTP traffic, so HTTPS links lead to an error complaining about lacking SSL credentials. I’ll have to address this, too, at a later date.
Anyway, I had to do some digging to find the problem. The HTTP link preview was good but somewhere along the line it was redirecting. Opening up the developer console with F12 gave much more insight into the problem.
For some reason content on the page was sending a GET request to HTTPS. I am familiar enough with GET requests but I still wasn’t sure what was sending it. Scrolling down a little more gave a clue.
The host and referer are good in the screenshot above. The more questionable header is “Upgrade-Insecure-Requests”. After a quick search, this header does exactly what its name says: if the request is using HTTP, turn it into HTTPS. This is a good security practice but the P101 network is currently not setup for it. Skimming through the Apache documentation looking for request header stuff lead me to a solution not long after. In the Apache VirtualHost file for the domain, I added the following directive:
RequestHeader unset Upgrade-Insecure-Requests
This line removes the Upgrade-Insecure-Requests header in the HTTP request so that it cannot look for HTTPS connections. Sure enough, everything started working!