Hi,
I've been doing what should be a dead simple file download using Apache HttpClient. However, for whatever reason, I hit a limit of about 5 MB for the file transfer before the connection gets reset. Anything up to that size, and this method works just fine. I've tested this on Content Server 16 on Windows Server 2012, and both Oracle 12 and SQL Server 2014 databases. I'm wondering if this is an inherit limit of Apache HttpClient but nowhere on Stack Overflow can I find anyone citing 5 MB as the upper limit (unless it's a function of memory). Here is my code:
public void getDocument(long nodeID, String downloadLocation, String filename) throws Exception {
CloseableHttpClient client = HttpClients.createDefault();
CloseableHttpResponse response = null;
boolean ok = true;
String errMsg = "";
InputStream is = null;
OutputStream os = null;
try {
// simply build a url of the form http://servername/OTCS/cs.exe/api/v2/nodes/<nodeID>/content
String url = getAPIURL(2).concat(String.format("nodes/%d/content", nodeID));
response = this.getResponseFromGet(client, url);
//System.out.println("Got response");
HttpEntity entity = response.getEntity();
//is = entity.getContent();
System.out.println("Got stream " + File.separator);
System.out.println("Writing to " + downloadLocation + File.separator + filename);
os = new FileOutputStream(new File(downloadLocation + File.separator + filename));
entity.writeTo(os);
System.out.println("Wrote to stream");
byte[] buffer = new byte[FILE_XFER_BUFF_SIZE];
int bytesRead = 0;
int passCount = 0;
int cumBytesRead = 0;
System.out.println("Begin xfer");
/*while((bytesRead = is.read(buffer)) > 0) {
os.write(buffer, 0, bytesRead);
cumBytesRead += bytesRead;
passCount++;
if (passCount % 10 == 0)
System.out.println(String.format("Pass %d, bytes read %d", passCount, cumBytesRead));
}*/
//is.close();
os.close();
System.out.println("Download complete");
} catch (Exception e) {
ok = false;
errMsg = e.getMessage();
} finally {
if (response != null)
response.close();
client.close();
if (!ok)
throw new RESTException(errMsg);
}
}
I've tried reading a fixed number of bytes into a WHILE block as well as just using inputStream.read() to outputStream.write(byte), but regardless of which of these approaches I take, I always hit a limit of about 5 MB. I need to scale this to at least 2 GB. On the upload side, I can easily get much more than that.
Any suggestions?
Thanks
-Hugh Ferguson
