Scanning and displaying every word from a website source code Java

**Lilac** · February 19th, 2014, 11:59 AM

I'm a new Computer Science student and I have been given a task to scan the contents of a websites source code, and use delimiters to extract all hyperlinks from the site and display them. We havent been told anything about how to do this so after some looking around online this is what I have so far:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Scanner;
 
public class HyperlinkMain {
	public static void main(String[] args) {
        try {
        	Scanner in = new Scanner (System.in);
        	String URL = in.next();
 
            URL website = new URL(URL);
            BufferedReader input = new BufferedReader(new InputStreamReader(website.openStream()));
            String inputLine; 
 
            while ((inputLine = input.readLine()) != null) {
                // Process each line.
                System.out.println(inputLine);
            }
            in.close(); 
 
        } catch (MalformedURLException me) {
            System.out.println(me); 
 
        } catch (IOException ioe) {
            System.out.println(ioe);
        }
    }
}

}

So my program can extract each line from the source code of a website and display it, but realistically I want it to extract each WORD as such from the source code rather than every line. I've looked around online but I don't really know how it's done because I keep getting errors when I use input.read();

Could anyone help me understand how to make it extract each word from the source code? Would be highly appreciated

**Norm** · February 19th, 2014, 12:09 PM

Can you post a small sample page and the "words" you want to extract from it?

The project sounds like it should read the lines of html text from the site and then parse the lines to extract the desired words.

**Lilac** · February 19th, 2014, 12:19 PM

Basically I will get given a web page and my program should be able to extract all the hyperlinks from that page
They said to use delimiters to extract the links, however I assume I need to extract every 'word' from the source code, then check using delimiters whether the 'word' is a link

**Norm** · February 19th, 2014, 12:59 PM

need to extract every 'word'

What is a "word"?
Can you post a small sample page and the "words" you want to extract from it?

**Lilac** · February 19th, 2014, 01:09 PM

Say the website was something like this:
<html>
<body>
<a href="www.google.com">GoogleLink</a>
</body>
</html>
How would I use a delimiter to detect the start of the link (<a href=") and the end of the link (")?

Ive changed my code around a bit:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Scanner;
import java.util.ArrayList;
 
 
public class HyperlinkMain {
	public static void main(String[] args) {
        try {
        	Scanner in = new Scanner (System.in);
        	String URL = in.next();
 
            URL website = new URL(URL);
            Scanner inWebsite = new Scanner (website.openStream());
 
            String inputLine; 
 
            while ((inWebsite.hasNext())) {
            	// Process each 'word'.
                System.out.println(inWebsite.next());
 
            }
            in.close(); 
 
        } catch (MalformedURLException me) {
            System.out.println(me); 
 
        } catch (IOException ioe) {
            System.out.println(ioe);
        }
    }
}

So all I need to do now is work out how to use a delimiter to find the start and end of the link

**Norm** · February 19th, 2014, 01:18 PM

If you are sure that there are no embedded spaces in the html, then the String class has methods to locate one String: "<a href=\"" within another String. The same method could be used to find the ending "\""

How would I use a delimiter to detect the start

Can you explain what that means? What is the delimiter you are talking about? Is this String: "<a href=\"" a delimiter?

**Lilac** · February 19th, 2014, 01:29 PM

I'm really new to programming so this isn't easy for me to explain or understand, but as the scanner goes through the source code, if it see's [<a href="] then it scans that until it finds [">] and outputs what should be between the '=' and the '>'

But i've been told to do this by setting delimiters, but I cant find much help on that

**Norm** · February 19th, 2014, 01:43 PM

Are you talking about the delimiters used by the Scanner class? They have to do with the Pattern class and regular expressions. I'm not very good with regular expressions.

**Lilac** · February 19th, 2014, 01:45 PM

Yes I believe that's along the right lines, using the scanner class: I understand that not many people use regular expressions which is why I'm finding it so hard to get help

**Norm** · February 19th, 2014, 01:58 PM

Would you be looking for something like this:
Given a stream of characters skip over characters until a match on the link tag, then match what follows up to the ending " character. The link tag has "<a" or "<A", One or More Spaces, "href", OMS, "=", OMS and "

**Lilac** · February 19th, 2014, 02:04 PM

That sounds like it could work, say if I the Scanner used the .next() function, it would go check it and if it finds <a href=" it then finds when it closes at "> and prints the link thats inside

**Norm** · February 19th, 2014, 02:07 PM

Now find someone that can write a regex to do that.

**Lilac** · February 19th, 2014, 02:10 PM

Thanks for trying to help me out on this, I appreciate your time

Thread: Scanning and displaying every word from a website source code Java

LinkBack

Thread Tools

Display

Scanning and displaying every word from a website source code Java

Related threads:

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Re: Scanning and displaying every word from a website source code Java

Similar Threads

Is there a java or php program where it searches for specific chosen code? say search engine codes but for website code specifically

Getting Value from Website source code.

How to Grab the HTML source code of a website URL index page?

How to Grab the HTML source code of a website URL index page?

Website Source