java Out of memory exception when downloading large files

HFCDev

Hi, 

  I am attempting to download some very large files for a data extraction. I have a Java application that reads through a spreadsheet, and gets info about the file to download. I'm using the Content Web Service to download so theoretically it should be chunking it out.

Here is the function with the line that stack traces hilighted:And here is the actual stack trace:

I am generating the ContentService client using the WSUtils class from the LLTree example where they set the chunk size, i.e

// instruct the connections to use chunked transfer encoding to support large files
        ( (BindingProvider) endpoint ).getRequestContext().put( JAXWSProperties.HTTP_CLIENT_STREAMING_CHUNK_SIZE, 1024000 );
    
        setSoapHeader( (WSBindingProvider) endpoint, token );
        

I tried setting this chunk size extremely small but that wasn't enough. I'm on a somewhat memory limited machine (4GB RAM) and the file is 32 MB in size.

Can anyone help???

-Hugh Ferguson

HFCDev

With a big shout out to Ossie Moore who pointed me to an obscure HTTP streaming method, here is a revised method that *does* work. I based it on the HTTP streaming upload example in the client docs, and created this replacement function:

public void downloadFileHttp(int uid, int versID, String filepath) throws Exception {
        URL     uri;
        String  contextID;
        String     token = getAuthenticationToken();
        HttpURLConnection  conn;

        Log("Attempting to get document context for " + uid);
        contextID = fDocMan.getVersionContentsContext(uid, versID);
        
        uri = constructURL(token, contextID, null, -1);
        // create the request with parameters specified as the query String
        // this request is a POST
        conn = (HttpURLConnection) uri.openConnection();

        // Setup the connection.
        conn.setDoOutput( true );
        conn.setDoInput( true );
        conn.setUseCaches( false );
        conn.setRequestMethod( "GET" );

        conn.connect();

        InputStream iStream =  conn.getInputStream();
        FileOutputStream fos = new FileOutputStream(filepath);
        byte[] buffer = new byte[BUFFER_SIZE];
        int bytesRead;
        int loopCtr = 0;
        while ((bytesRead = iStream.read(buffer, 0, BUFFER_SIZE))>0) {
            loopCtr++;
            fos.write(buffer, 0, bytesRead);
            if (loopCtr % 10 == 0) {
                fos.flush();
                //System.out.println("Flushing FOS");
            }
            //System.out.println("Loop ctr " + loopCtr);
        }
        fos.close();
        iStream.close();
        
        conn.disconnect();

    }

 

The constructURL() method is as follows (also taken from the documentation):

private URL constructURL(
            String    token,
            String    contextID,
            String    fileName,
            long    fileSize )    throws Exception {
        String uri = baseURL.replace("Authentication.svc", "HttpContentHandler.ashx");

        uri += "?token=" + URLEncoder.encode( token, "utf-8" ) + "&contextID=" + contextID;

        if ( fileName != null )
        {
            uri += "&fileName=" + fileName + "&fileSize=" + String.valueOf( fileSize );
        }

        return new URL( uri );

    }

 

Once again, many thanks to Ossie Moore for pointing me in this direction!

-Hugh