Thread: Stop word removal and stemming

ideas on how to program in java

What parts of your program design are you having problems with coding in java?
Is your problem with designing a computer program
or with coding a program design in java?

input file for reading

One of the easy classes for reading a file is the Scanner class. It has many useful methods that allow you to read the file in different ways.

how to find the stop words and stem

The String class has many methods for searching a String.

i'm having trouble with coding.

can you describe in more detail what you are trying to code?
Also an example might help. Post some text that contains what you are trying to find and describe what in that text you want to find.

Is this what you are trying to do:
Take a sentence of words, separate the words and assign some tag to each word.
In your example, the tags were the UPPERCASE letters after each word.

"This is a short sentence "
This DT
is VBZ
short JJ
sentence NN.

DT is the tag for the word: "This"

Where do the tags come from?

See the Scanner class for methods that can be used to read data from a single file.
If you want to read all the files in a folder, you need to use the File class to get a list of the files in the folder and then use the Scanner class to read each file in the list.

this is the code i have for stop word removal. can you help me to read the data from files in a folder istead of directly giving it in the program

package split;
 
import java.util.*;
import java.io.*;
public class Split {
 
public static void main(String[] args)
{
for(String word : Split.words("The rain in spain falls mainly on the plains, except when it's not exactly working that way! And I need+some= way. How~ will \"(this)\" \"work\"?"))
System.out.println(word);
 
 
}
 
static HashSet stopwords = new HashSet();
 
public static void addStopwords()
{
try{
BufferedReader br = new BufferedReader(new FileReader("stopwords.txt"));
 
while(br.ready())
{
stopwords.add(br.readLine());
}
 
}
catch(Exception e){System.out.println(e);}
}
 
public static ArrayList<String> words(String line)
{
if(stopwords.size() == 0)
addStopwords();
 
ArrayList result = new ArrayList();
 
String[] words = line.split("[ \t\n,\\.\"!?$~()\\[\\]\\{\\}:;/\\\\<>+=%*]");
for(int i=0; i < words.length; i++)
{
if(words[i] != null && !words[i].equals(""))
{
String word = words[i].toLowerCase();
if(!stopwords.contains(word))
{
result.add(Stemmer.stem(word));
}
}
}
 
return result;
}
 
}

can you help me to read the data from files in a folder

I see that the code is using the BufferedReader class's methods to read a file.
What problems are you having reading the data?

Please edit your post and wrap the code with code tags. Be sure the code is properly formatted. Nested statements should be indented. All statements SHOULD NOT start in the first column.

Did you miss post#8?

The File class has methods that return a list of the files in a folder. You can use that list to read the files in the folder one at a time.

The steps are:
get a list of the files in the folder
begin loop
get next file in the list
read data from that file
end loop

The system cannot find the file

The program can not find the file in the location where you are looking. To find where the program is looking for the file, create a File object for the file you are trying to read and print that File object's absolute path value which will show you where the program is looking for the file.

I don't know what your IDE does with the location of files when a program tries to read a file.

See the Scanner class for methods to read data from a file. There are many examples of code using the Scanner class's methods here on the forum. Do a Search.

What have you tried? What problems are you having with it?