Thread: Web Crawling is too Slow!

Hello Sir,

my code (pasted below) working properly but at very slow rate. can you please suggest any changes required (if any) from programming side to make to work fast. thanks in advance.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.net.PasswordAuthentication;
import java.net.*;
import java.io.*;
import java.lang.Object;
 
class ProxyAuthenticator extends java.net.Authenticator {
 
    private String user, password;
 
    public ProxyAuthenticator(String user, String password) {
        this.user = user;
        this.password = password;
    }
 
    protected java.net.PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(user, password.toCharArray());
    }
 
}
 
public class CrawlWeb1 {
 
	public static void main(String args[]) {
		try {
		  	Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(<IPAddr>, <Port>));
      			Authenticator.setDefault(new ProxyAuthenticator(<username>, <password>));
			String surl = "http://www.timesofindia.com/";
			URL asksearch = new URL(surl);
			URLConnection yc = asksearch.openConnection(proxy);
			BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
			String inputLine;
			FileWriter fw = new FileWriter("timeofindia.html");
			BufferedWriter out = new BufferedWriter(fw);
			Boolean body = false;
			while ((inputLine = in.readLine()) != null) {
				out.write(inputLine);
			}
			System.out.println("Crawling Done...! URL : " +surl);
			out.close();
			fw.close();
			in.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

That is essentially as fast as that code is going to be (other than possibly a few insignificant optimizations).

The only way I can see this code running faster is if your network speed gets faster, or if the website you're connecting to decides to allocate more bandwidth for you. Whichever is the slower of the two will limit what your actual download speed is.

I don't really think much can be done from the programming side. The code looks OK to me. I think the speed will totally depend on the internet connection and server response time.