I'm trying to use java.security to decrypt a 256-byte hexademical message encrypted with RSA as part of a college assignment but I've encountered problems when it comes to converting the hex to a byte array and passing it to a Cipher object.

I think the problem may have something to do with padding and the conversions between formats that I'm doing and since I don't really know much about dealing with bytes I've brickwalled after trying what I could think of and Googling for the answer.

I load up the modulus and exponent of a previously generated private key and generate a private key using those, which I'm pretty sure is working or is at least not the problem now.
Then I read in the 256-byte hex string from a file and convert it to decimal (which I'm pretty sure is the form java.security always expects). That seems to go perfectly fine.
Then, because the Cipher object expects a byte array to decrypt in its doFinal() method, I convert the decimal ciphertext into a byte array and I lose confidence in what's going on from there.

Here's the code snippet:

//Load and decrypt cipher text
	           BufferedReader ctReader = new BufferedReader( new FileReader( cipherTextFile ) );
	           String cText = ctReader.readLine();
	           BigInteger cipherBI = new BigInteger( cText, 16 );
	           cText = cipherBI.toString();
	           System.out.println( "cText(Dec): " + cText );
 
	           byte[] cTextBytes = cipherBI.toByteArray();
 
	           Cipher cipher = Cipher.getInstance( "RSA" );
	           cipher.init( Cipher.DECRYPT_MODE, prKey );
	           byte[] cipherData = cipher.doFinal( cTextBytes, 0, 256 );

I read the ciphertext from the file, convert the hex string to a BigInteger and then create a byte array and set it to the BigInteger converted to a byte array.

I can't see where I went wrong there but after that, I get confused.
I initialize a Cipher with the "RSA" algorithm in DECRYPT_MODE with the previously loaded private key.
Then I call cipher.doFinal() on the byte array and store the result in another byte array which should be the decrypted message. The 2 other arguments are the input offset and length, which I set to 256.

When I run the program, I get an IllegalArgumentException on that last line, so I'm assuming there's something wrong with the byte representation of the ciphertext.

I'm thinking that maybe the conversion from hex to decimal and then to a byte array may have altered the length perhaps by eliminating zeros and so it's no longer 256 bytes and maybe that's the problem but I don't know. Also I thought maybe the algorithm was wrong so I tried using "RSA/ECB/NoPadding" but I get the same result and I'm really not entirely sure which I should using. Maybe I need to pad the byte array somehow?

Does anyone here have any experience with RSA and java.security? Or maybe I'm just making a general mistake converting a hex string to a byte array?