Thread: Splitting large video files into small chunks and combining them

Hi All,

Thank you for viewing my thread. I have searched for hours but cannot find any information on this topic.

I need to upload very large video files. The files can be something like 10GB in size, currently .mpeg format but they may be .mp4 format later on too.

Since the files will be uploaded remotely using using internet from the clients mobile device (3G or 4G in UK) I need to split the files into smaller chunks. This way if there is a connection issue (very likely as the files are large) I can resume the upload without doing the whole 10gb again.

My question is, what can I use to split a large file like 10Gb into smaller chunks like 200mb, uplaod the files and then combine them again? Is there a jar file that I can use. i cant install stuff like ffmpeg on the clients laptops.

Thank you for your time.

Kind regards,
Imran

Hi Norm,

Thank you for replying to my thread. Can you point in the right direction on what Java classes/api I can use to split and merge video files please? I ahve looked at Apache Commons but cannot find if its possible.

Kind regards,
Imran

Hi Norm and Aussiemcgr,

Thank you for getting back. Its much more clearer to me know. I will convert the video file into a byte array and then convert it back to the video. I will give this a try and report back in a few days to let you know how I get on.

Kind regards,
Imran

Hi,

Ive been reading up on socket programming in Java. What do you think about sending multiple byte[] array's over a socket compared with sending byte files over ftp?

I need a way to resume the upload if there is a connection issue.
With the byte files approach I can do a checksum of the files on both sides to make sure its moved over without issues. I can also keep track on how many files are moved over in the db so I can resume if there was a issue. The problem with this approach is that its going to create loads and loads of small files for a 10gb file.

Im not sure how I would keep track using sockets? Do you think this may be a better approach?

I basically trying to replicate a dropbox like feature.

Thank you for your time,
Imran

Thanks for getting back.

Im trying to make it easier for myself and use FTP4J which supports restart. But I cannot get it to work. It works fine when uploading the whole file. But when I stop it and provide it a offset (number of bytes that was transferred) it just hangs. No errors.

Here is my code:

 
import it.sauronsoftware.ftp4j.FTPClient;
import java.io.File;
 
public class FTP4J {
 
	private static String server = ""; 
	private static String username = ""; 
	private static String password = "";
 
	private String local_dir = "";
	private String remote_dir = ""; 
 
    public static void main(String[] args) {    	
    	String filename = "small.mp4";
    	boolean append = true;
    	(new FTP4J()).uploadFile(filename, append);
    }
 
    public void uploadFile(String filename, boolean appendIt) {    	
        FTPClient client = null;
        try {
        	client = new FTPClient();
        	client.connect(server);
        	client.login(username, password);
 
        	//if(client.isResumeSupported()) {
        		//System.out.println("Resume is supported");
        	//}
 
             if (client != null) {
                try {
                    client.setPassive(true);	
                    client.setType(FTPClient.TYPE_BINARY);
                    client.changeDirectory(remote_dir);
 
                    String filePath = local_dir+filename;                      
                    File fileUpload = new File(filePath);
 
                    //Upload the file
                    if(appendIt) {
                    	System.out.println("Appending the " + filePath + " to Remote Machine");
                    	long offset = 3803616; // taken from server
                    	client.upload(fileUpload, offset, new MyFTPListener());
                    } else {
                    	System.out.println("Uploading the " + filePath + " to Remote Machine");
                    	client.upload(fileUpload, new MyFTPListener());
                    }
 
                    System.out.println("Successfully Uploaded the " + filePath + " File to Remote Machine");
                } catch (Exception e) {
                        System.err.println("ERROR : Error in Transfering File to Remote Machine");
                        System.exit(3);
                }
            }
        } catch (Exception e) {
            System.err.println("ERROR : Error in Connecting to Remote Machine");
            System.exit(1);
        }
 
        finally {
            if (client != null) {
                try {
                    client.disconnect(true);
                }
                catch (Exception e) {
                    System.err.println("ERROR : Error in disconnecting the Remote Machine");
                }
            }
        }
        System.exit(0);
    }

Please help me find the issue. Many thanks

Kind regards,
Imran

Thanks for getting back.

This is what the print statements print. After line 3 it hands for like 15 minutes. I have looked into breaking the big files but the issue I have with that is with a 10gb file it will create 100000 files, 100mb each. Thats a lot of files. I guess I can create one file at a time move it over.

1. Resume is supported
2. Appending the /Users/adamm001/Documents/Projects/ADaMM CCTV Uploader/upload/small.mp4 to Remote Machine
3. TRANSFER-STATUS: File transfer started...
4. TRANSFER-STATUS: File transfer failed...
5. ERROR : Error in Transfering File to Remote Machine

I have created the following listener class which the above code uses:

package controller;
 
import it.sauronsoftware.ftp4j.FTPDataTransferListener;
 
public class MyFTPListener implements FTPDataTransferListener
{
 
        public void started()
        {
                // Transfer started
                System.out.println("TRANSFER-STATUS: File transfer started...");
        }
 
        public void transferred(int length)
        {
                // Yet other length bytes has been transferred since the last time this
                // method was called
        }
 
        public void completed()
        {
                // Transfer completed
                System.out.println("TRANSFER-STATUS: File transfer completed...");
        }
 
        public void aborted()
        {
                // Transfer aborted
                System.err.println("TRANSFER-STATUS: File transfer aborted...");
        }
 
        public void failed()
        {
                // Transfer failed
                System.err.println("TRANSFER-STATUS: File transfer failed...");
        }
}

Kind regards,
Imran

This is what I get:

Resume is supported
Appending the /Users/adamm001/Documents/Projects/ADaMM CCTV Uploader/upload/small.mp4 to Remote Machine
TRANSFER-STATUS: File transfer started...
TRANSFER-STATUS: File transfer failed...
it.sauronsoftware.ftp4j.FTPException [code=451, message= small.mp4: Append/Restart not permitted, try again]
at it.sauronsoftware.ftp4j.FTPClient.upload(FTPClient .java:2794)
at it.sauronsoftware.ftp4j.FTPClient.upload(FTPClient .java:2586)
at controller.FTP4J.uploadFile(FTP4J.java:68)
at controller.FTP4J.main(FTP4J.java:19)