Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 10 of 10

Thread: java crawler

  1. #1
    Junior Member
    Join Date
    May 2013
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default java crawler

    salamzz my final year project is java crawler nd em confused related to starting plzz help me..which one is the best algo for the crawler


  2. #2
    Member angstrem's Avatar
    Join Date
    Mar 2013
    Location
    Ukraine
    Posts
    200
    My Mood
    Happy
    Thanks
    9
    Thanked 31 Times in 29 Posts

    Default Re: java crawler

    Consider this course. They'll teach you the basics of programming on Python on a web crawler example.

  3. #3
    Junior Member
    Join Date
    May 2013
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: java crawler

    bt python is different language and i have to make web crawler in java bt cn't understand frm where i start my work

  4. #4
    Member angstrem's Avatar
    Join Date
    Mar 2013
    Location
    Ukraine
    Posts
    200
    My Mood
    Happy
    Thanks
    9
    Thanked 31 Times in 29 Posts

    Default Re: java crawler

    Doesn't matter. They will show you an algorithm and explain it, so that you know where to start. Crawler is a trivial project, who can be implemented just in one simple function, and if you know the material you were supposed to learn this year, you'll definitely be able to do it.

  5. #5
    Junior Member
    Join Date
    May 2013
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: java crawler

    okhzz thnkss sir can you send m the useful link frm where i gt the usefull inf..r which algo is best for the crawler

  6. #6
    Member angstrem's Avatar
    Join Date
    Mar 2013
    Location
    Ukraine
    Posts
    200
    My Mood
    Happy
    Thanks
    9
    Thanked 31 Times in 29 Posts

    Default Re: java crawler


  7. #7
    Junior Member
    Join Date
    May 2013
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: java crawler

    one of my uni sir said use specificc website frm where u want to access data bt other teachers said k thats nt possible.r i js want relevent data nt other stuff frm the otherweb sites..

  8. #8
    Member angstrem's Avatar
    Join Date
    Mar 2013
    Location
    Ukraine
    Posts
    200
    My Mood
    Happy
    Thanks
    9
    Thanked 31 Times in 29 Posts

    Default Re: java crawler

    The relevancy can be achieved by using filtering function during graph traversal.

  9. #9
    Junior Member
    Join Date
    Sep 2013
    Posts
    11
    My Mood
    Bored
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default Re: java crawler

    Hi,
    You can write a crawler using java with the help of existing API’s. You can use the below algorithm to start you java crawler final year project. while(list of unvisited URLs is not empty) {
    take URL from list
    fetch content
    record whatever it is you want to about the content
    if content is HTML {
    parse out URLs from links
    foreach URL {
    if it matches your rules
    and it's not already in either the visited or unvisited list
    add it to the unvisited list
    }
    }
    }

  10. #10
    Super Moderator
    Join Date
    Jun 2013
    Location
    So. Maryland, USA
    Posts
    5,517
    My Mood
    Mellow
    Thanks
    215
    Thanked 698 Times in 680 Posts

    Default Re: java crawler

    Please, please, please, don't resurrect dead threads. Check the date the thread was started AND the date of the last post. If they're not within the past 2 or 3 weeks, the thread isn't relevant anymore.

    Closing this one.

Similar Threads

  1. Crawler has trouble logging in.
    By bholzer in forum What's Wrong With My Code?
    Replies: 0
    Last Post: November 13th, 2011, 06:59 PM
  2. Threads in Web Crawler
    By relion65 in forum Threads
    Replies: 5
    Last Post: June 28th, 2011, 11:48 PM
  3. Crawler is getting slow
    By manokrrish in forum Threads
    Replies: 3
    Last Post: October 18th, 2010, 09:24 AM
  4. How do we navigate to a page with the anchor link in Web Crawler program?
    By expertOpinion in forum What's Wrong With My Code?
    Replies: 5
    Last Post: September 10th, 2009, 04:44 PM