Upload Large Files ~Upto 50GB

Phaneendra Duggirala

Hi,

I'm building a .net Web application with a requirement to upload a large file upto 50GB. As of Today max. file size is 5 GB but it will grow as big as 50 GB down the time. These are some application specific files.

I am trying to see if I can achieve this by dividing the file into chunks and upload them. Below is the code I'm working on but it doesnt help. Let me know if there is any solution to this or anyone has achieved uploading this much big files.

Thanks in advance.  

public void Upload2(string AuthToken) { otAuth.AuthenticationToken = AuthToken; string name = "New Document111_" + System.DateTime.Now.Ticks;   string FilePath = @C:\Users\TestUser\Desktop\HugeFileRollforward\5GBFilesZip.zip;   int Offset = 0; int ChunkSize = 65536; byte[] Buffer = new byte[ChunkSize]; FileStream fs = new FileStream(FilePath, FileMode.Open, FileAccess.Read); FileInfo fileInfo = new FileInfo(FilePath);   long FileSize = new FileInfo(FilePath).Length; fs.Position = Offset; int BytesRead = 0; Node uploadedFile = null;   try { while (Offset != FileSize) { BytesRead = fs.Read(Buffer, 0, ChunkSize); if (BytesRead != Buffer.Length) { ChunkSize = BytesRead; byte[] TrimmedBuffer = new byte[BytesRead]; Array.Copy(Buffer, TrimmedBuffer, BytesRead); Buffer = TrimmedBuffer; } try { fs.Seek(Offset, SeekOrigin.Begin); Attachment attachment = new Attachment(); attachment.Contents = Buffer; attachment.CreatedDate = fileInfo.CreationTime; attachment.FileName = fileInfo.Name; attachment.FileSize = Buffer.Length; attachment.ModifiedDate = fileInfo.LastWriteTime;   if (uploadedFile == null) { uploadedFile = docManClient.CreateDocument(ref otAuth, 2000, name, "Uploaded from POC webservice", false, null, attachment); } else { uploadedFile = docManClient.CreateDocument(ref otAuth, uploadedFile.ID, name, "Uploaded from POC webservice", false, null, attachment); //I’m trying to update the content byte to the existing node. But I get error here that I cannot attach a file to a document container. It makes sense. But is there any other solution? }     } catch { break; }     // Offset is only updated AFTER a successful send of the bytes. Offset += BytesRead; // save the offset position for resume } } catch (Exception ex) { } finally { fs.Close(); } }

 

eLink User

Hi Phaneendra,
You may want to read up on the first thread pinned on the forum.If your WS is hosted on IIS I think that web server will allow only a maximum of 4 GB (IIS7/7.5/8) but what I want you to know is the webserver Web.Config and the App.Config has to be corrected.Kyle Swidrowich has written a beautiful snippet there.Once you have played and understood the Kyle code then start doing what you really want. In the WSAPI documentation you would see OT saying the HTTP handler mechanism.There is a lot of documentation regarding the MTOM  and http handler

Once uploaded to livelink how would you get the 50GB file from livelink,thru the webgui or some other mechanism.50 GB files in livelink seems to be very huge.Surely the search indexer might choke on that file...

Sorry I have no experience in dealing with such huge files so check with OT prod Management and see if they can handle these files or not...

..Appu

Never argue with idiots. They drag you down to their level, and then beat you with experience.
Eat one live toad the first thing in the morning and nothing worse will happen to you the rest of the day

via : Laugh IT Out: Dilbert’s Best Quotes

On Thu, Jun 12, 2014 at 7:25 AM, eLink Entry: Content Web Services Forum <otdncontentwebservices971forum@elinkkc.opentext.com> wrote:

Upload Large Files ~Upto 50GB

Posted byphduggirala@deloitte.ca (Duggirala, Phaneendra)On 06/12/2014 08:23 AM

Hi, I'm building a .net Web application with a requirement to upload a large file upto 50GB. As of Today max. file size is 5 GB but it will grow as big as 50 GB down the time. These are some application specific files. I am trying to see if I can achieve this by dividing the file into chunks and upload them. Below is the code I'm working on but it doesnt help. Let me know if there is any solution to this or anyone has achieved uploading this much big files. Thanks in advance.    public void Upload2(string AuthToken) { otAuth.AuthenticationToken = AuthToken; string name = "New Document111_" + System.DateTime.Now.Ticks;   string FilePath = @C:\Users\TestUser\Desktop\HugeFileRollforward\5GBFilesZip.zip;   int Offset = 0; int ChunkSize = 65536; byte[] Buffer = new byte[ChunkSize]; FileStream fs = new FileStream(FilePath, FileMode.Open, FileAccess.Read); FileInfo fileInfo = new FileInfo(FilePath);   long FileSize = new FileInfo(FilePath).Length; fs.Position = Offset; int BytesRead = 0; Node uploadedFile = null;   try { while (Offset != FileSize) { BytesRead = fs.Read(Buffer, 0, ChunkSize); if (BytesRead != Buffer.Length) { ChunkSize = BytesRead; byte[] TrimmedBuffer = new byte[BytesRead]; Array.Copy(Buffer, TrimmedBuffer, BytesRead); Buffer = TrimmedBuffer; } try { fs.Seek(Offset, SeekOrigin.Begin); Attachment attachment = new Attachment(); attachment.Contents = Buffer; attachment.CreatedDate = fileInfo.CreationTime; attachment.FileName = fileInfo.Name; attachment.FileSize = Buffer.Length; attachment.ModifiedDate = fileInfo.LastWriteTime;   if (uploadedFile == null) { uploadedFile = docManClient.CreateDocument(ref otAuth, 2000, name, "Uploaded from POC webservice", false, null, attachment); } else { uploadedFile = docManClient.CreateDocument(ref otAuth, uploadedFile.ID, name, "Uploaded from POC webservice", false, null, attachment); //I’m trying to update the content byte to the existing node. But I get error here that I cannot attach a file to a document container. It makes sense. But is there any other solution? }     } catch { break; }     // Offset is only updated AFTER a successful send of the bytes. Offset += BytesRead; // save the offset position for resume } } catch (Exception ex) { } finally { fs.Close(); } }

 

---

[To post a comment, use the normal reply function]
Forum:	Content Web Services Forum
Content Server:	Knowledge Center