Thread: ChunkedInputStream Error? Help Please

Hey guys,
I have made an application that reads of a website and parses it's HTML, however when i go through more than one page or so(maybe 2-3) its throws out an error. This only happens when i parse lots of pages at once...

This is the Error:

java.io.IOException: Premature EOF	
        at sun.net.[url]www.http.ChunkedInputStream.readAheadBlocking(Unknown[/url] Source)
	at sun.net.[url]www.http.ChunkedInputStream.readAhead(Unknown[/url] Source)
	at sun.net.[url]www.http.ChunkedInputStream.read(Unknown[/url] Source)
	at java.io.FilterInputStream.read(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.BufferedReader.fill(Unknown Source)
	at java.io.BufferedReader.readLine(Unknown Source)
	at java.io.BufferedReader.readLine(Unknown Source)
	at Youtube1.main(Youtube1.java:40)

Line 40 is:

BufferedReader br = new BufferedReader(new InputStreamReader(is));

If you know how to help and would like to i can PM you the full code.


Thanks for the help guys!

PS.
I'm kinda new to java and this is my first project.

We need to see more code. A "Premature EOF" (end of file) means that the file ended (for example, the sender stopped sending it) before the reciever (your code) expected it.

Can you show a SSCCE?

I'll post the whole code, here:

import java.util.*;
import java.net.*;
import java.io.*;
import javax.swing.JOptionPane;
 
public class Youtube{
   public static void main(String args[]){
 
	  String video;
      String comment;
      int number = 0;
      InputStream is = null;
      String line;
      URL page;
      boolean check = false;
      int namepos1;
      int namepos2;
      int reply;
      List<String> names = new ArrayList<String>();     
 
      video = JOptionPane.showInputDialog("Please Enter The First Youtube Comment Page URL:", "http://www.youtube.com/all_comments?v=******");
 
      String Scomments = JOptionPane.showInputDialog("Please Enter The Total Number Of Comments:", "500");
 
      double comments = Double.parseDouble(Scomments);
 
      int pages = (int) Math.ceil( comments /  500d );
      number++;      
      comment = video + "&page=" + number;
 
         try{
           while(number <= pages){
           	    comment = video + "&page=" + number;
        	    number++;                       
                page = new URL(comment);                
                is = page.openStream(); 
                BufferedReader br = new BufferedReader(new InputStreamReader(is));
 
                while ((line = br.readLine()) != null){       
                    if(check){
                    namepos1 = line.indexOf("yt-user-name ");
                    namepos2 = line.lastIndexOf("<");
                    reply = line.indexOf("in reply to");
 
                    if(namepos1 > 0 && namepos2 > 0 && reply < 0){
                       int back=line.lastIndexOf("<");
                       int front=line.indexOf("yt-user-name ") + 25;
                       String aaa=line.substring(front , back);
                       if(!names.contains(aaa)) names.add(aaa);  
                       System.out.println("Found Username");
                    }else{}
 
                    check = false;                                       
                    }                    
                    if(line.indexOf("author ") != -1) check = true;
 
                  } 
                  is.close();
     } 
           System.out.println("Done. Loading Unique UserNames...");
 
          }catch (MalformedURLException mue) {
                mue.printStackTrace();
          } catch (IOException ioe) {
               ioe.printStackTrace();
          }catch(Exception e){
               e.printStackTrace();
         }
 
            System.out.println("\n \n");
            System.out.println("List Of Unique Commenter Usernames:");
            for(String name: names){
               System.out.println(name);
            }
 
 
       Random r = new Random();
       int winner = r.nextInt(names.size());
       JOptionPane.showMessageDialog(null, "The Random Commenter Username Is:   " + names.get(winner), "Random Commenter", JOptionPane.PLAIN_MESSAGE );
       System.out.println("\n\n\nThe Random Commenter Username Is:   " + names.get(winner));
 
 
 
 
    }
 
}

Basically You enter a Youtube comment page URL, and tell the program how many comments you want it to go through.

Then it'll grab all the usernames and add them to a list except duplicates. Then generate a random winner.

- for youtube give aways.

500 comments per page so i can run it up to 1000 sometimes 1500 but after that its pauses for ages and gives me an error.

Geez, if I was YouTube I'd kick your program out too. I'm surprised you got this far. The problem is that you're not talking HTTP - you're thinking in terms of a raw Socket stream. You can't just keep requesting data and expect the server to give it to you. You've got to follow the protocol. Additionally, you're making your life horrible by trying to parse HTML through code.

Lastly, you're breaking the YouTube terms of service:

You agree not to access Content through any technology or means other than the video playback pages of the Service itself, the Embeddable Player, or other explicitly authorized means YouTube may designate.

So, where do you go from here? Your best bet is to use the Google Java client library that includes a small YouTube client to access the data. This hides many of the details of dealing with the HTTP protocol and doesn't violate any terms of service.

In short - I'd rethink your design. Ultimately I'm guessing that YouTube has something to govern the kind of abuse you're throwing at their servers and/or there is something HTTP is telling you that you're ignoring. This may be a Cookie, a redirect, or something else.

Hmmm, i looked into the API, Just one question, it only displays the comments off the first video page, using the video ID i can only get these comments. How would i display all the comments? or navigate to each page?



Thanks.

Thanks for the help voodoo i kinda understand it more.

Originally Posted by Voodoo

public class ChunkedInputStream extends InputStream {
    private boolean isahead  = false;
    private StringBuffer buf;
    private InputStream in;
    private int ahead = -1;
    private int clen, ch;
    private boolean eof;
    //
    public ChunkedInputStream(InputStream in) throws IOException {
        this.in  = in;
        getChunk();
    }
    //
    protected boolean getChunk() throws IOException {
    	buf = new StringBuffer();
        while (true) {
        	ch = in.read();
        	if (ch == -1) throw new IOException("Premature end of chunked stream.");
        	else if (ch > 0x20 && ch < 0x7F) buf.append((char)ch);
        	else if (ch == '\n') break;
        	else if (ch == '\r') {
        		if ((ch = in.read()) != '\n') {
        			ahead = ch;
        			isahead = true;
        		}
        		break;
        	}
        }
        // Parse the buffer content as an hex number:
        char c;
        clen = 0;
        int fac = 1;
        int len = buf.length();
        while (--len >= 0) {
        	c = buf.charAt(len);
        	ch = (c > '9') ? 9 + (c & 0x0F) : (c & 0x0F);
        	if (ch > 15) break;
            	clen += (fac * ch);
            	fac *= 16;
        }
        eof = (clen == 0);
        return eof;
    }
    //
    protected boolean nextChunk() throws IOException {
        if (eof) return true;
        ch = in.read();     // '\r'
        ch = in.read();     // '\n'
        return getChunk();
    }
    //
    public int read()  throws IOException {
        if (clen == 0) {
            if (nextChunk()) return -1;
            if ( isahead ) {
                --clen;
                isahead = false;
                return ahead;
            }
        } 
        --clen;
        return in.read();
    }
}

That's the C.I.S.java, and this is my Youtube.java:

import java.util.*;
import java.net.*;
import java.io.*;
import javax.swing.JOptionPane;
 
public class Youtube{
   public static void main(String args[]){
 
	  String video;
      String comment;
      int number = 0;
      InputStream is = null;
      String line;
      URL page;
      boolean check = false;
      int namepos1;
      int namepos2;
      int reply;
      List<String> names = new ArrayList<String>();   
 
      //user inputs
      video = JOptionPane.showInputDialog("Please Enter The First Youtube Comment Page URL:", "http://www.youtube.com/all_comments?v=******");
 
      String Scomments = JOptionPane.showInputDialog("Please Enter The Total Number Of Comments:", "500");
 
      //calculates number of comment pages
      double comments = Double.parseDouble(Scomments);
 
      int pages = (int) Math.ceil( comments /  500d );
      number++;      
      comment = video + "&page=" + number;
 
      //goes through each page grabbing the user names and adding unique names to list
         try{
           while(number <= pages){
           	    comment = video + "&page=" + number;
        	    number++;                       
                page = new URL(comment);                
                is = page.openStream(); 
                BufferedReader br = new BufferedReader(new InputStreamReader(is));
 
 
                while ((line = br.readLine()) != null){       
                    if(check){
                    namepos1 = line.indexOf("yt-user-name ");
                    namepos2 = line.lastIndexOf("<");
                    reply = line.indexOf("in reply to");
 
                    if(namepos1 > 0 && namepos2 > 0 && reply < 0){
                       int back=line.lastIndexOf("<");
                       int front=line.indexOf("yt-user-name ") + 25;
                       String aaa=line.substring(front , back);
                       if(!names.contains(aaa)) names.add(aaa);  
                       System.out.println("Found User Name");
                    }else{}
 
                    check = false;                                       
                    }                    
                    if(line.indexOf("author ") != -1) check = true;
 
                  } 
                  is.close();
     } 
 
           //finished finding
           System.out.println("Done. Loading Unique User Names...");
 
          }catch (MalformedURLException mue) {
                mue.printStackTrace();
          } catch (IOException ioe) {
               ioe.printStackTrace();
          }catch(Exception e){
               e.printStackTrace();
         }
             //prints list
            System.out.println("\n \n");
            System.out.println("List Of Unique Commenter User Names:");
            for(String name: names){
               System.out.println(name);
            }
 
          //generates winner
       Random r = new Random();
       int winner = r.nextInt(names.size());
       JOptionPane.showMessageDialog(null, "The Random Commenter User Name Is:   " + names.get(winner), "Random Commenter", JOptionPane.PLAIN_MESSAGE );
       System.out.println("\n\n\nThe Random Commenter User Name Is:   " + names.get(winner));
 
 
 }
 
}

how and where do i implement the method read() and do the input streams have to be the same?