While it is easy to get this information from unsecured HTTP resources it's often the case that such a resource is only available for secured access through HTTP.
In that case and to ensure that this connection is trusted you have to check the server certificate for that connection.
Some sample piece of code to access and download data over http can look like this one:
string data = String.Empty; using (WebClient webClient = new WebClient()) { byte[] rawData = webClient.DownloadData(new Uri(/* path to resource */)); using (MemoryStream memStream = new MemoryStream(rawData)) { using (StreamReader reader = new StreamReader(memStream, true)) { data = reader.ReadToEnd(); } } }
This code works well for connections through HTTP.
Now you have to access the same resource over HTTPS and this is where you have to face a new challenge:
Validation of Certificates.
In the case of .NET there is the ServicePointManager that let you intercept the mechanism of validation.
It defines the ServerCertificateValidationCallback method which is called on secure resource access.
The adjusted code is now something like this:
string data = String.Empty;
using (WebClient webClient = new WebClient()) { RemoteCertificateValidationCallback validationCallback = (sender, cert, chain, errors) => { return true; }; ServicePointManager.ServerCertificateValidationCallback += validationCallback; try { byte[] rawData = webClient.DownloadData(new Uri(/* path to resource */)); using (MemoryStream memStream = new MemoryStream(rawData)) { using (StreamReader reader = new StreamReader(memStream, true)) { data = reader.ReadToEnd(); } } } finally { ServicePointManager.ServerCertificateValidationCallback -= validationCallback; } }
This code simply declares all certificates as valid which is some kind of stupid.
Let's focus on the validation of the certificate. Feel free to implement you own code of validation or just simply use .NET integrated features.
Something I did at bugtraqplugin is to use the default validation chain which depends on certificates stored at the Windows certification storage.
My simple approach:
RemoteCertificateValidationCallback validationCallback =
(sender, cert, chain, errors) => { return (certificate as X509Certificate2 ?? new X509Certificate2(certificate)).Verify(); };
All code put together can be available at source code of bugtraqplugin.