OD Failures on Transient Files

KenS

Hi All.

I have developers who use sftp(scp) to stage files to a central directory for deployment to our web servers (we're running OpenDeploy 6.1.1 on Solaris 9).

SSH creates transient files (usually 13 digits) that get renamed to the source file name after the copy is complete:

Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812'
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812'
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812'
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Uploading `/1235660859812' (flags creat )
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Closed file `/1235660859812' (handle=ba68d)
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] File `/1235660859812' closed (written: 3.1E, read: 7.0E).
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Setstat file `/1235660859812' (perms)
Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Renaming `/1235660859812' to `/consumer_symalllanguages.zip'

(Hostnames deleted & directory structure modified for brevity sake)

If a transactional based, directory comparison starts at the same time, it will see the transient file as new content, and will prepare to send it as a part of the deployment. However, when the actual transfer phase of the deployment starts, the transient file will have been "renamed", so to OD the transient file no longer exists, and OD fails the job, then rolls back the content.

And even though this transient file doesn't exist long, the frequency of this type failure is more common than you'd imagine when hundreds of files are being staged for publishing each day, and multiple jobs are deployed, manually or automatically.

I could try to create an inclusion Filter, and am building a list of extensions for that purpose.

Any performance issues if this list grows to more than [5|10|20?] different extensions?

I could try an exclusion filter, looking for any 13-character all-numeric files; is OD's regex reliable for that?

Has anyone else experienced this; and/or can you give any advice on how to prevent it from recurring?

Any help is appreciated,

KenS

Migrateduser

Hi All. I have developers who use sftp(scp) to stage files to a central directory for deployment to our web servers (we're running OpenDeploy 6.1.1 on Solaris 9). SSH creates transient files (usually 13 digits) that get renamed to the source file name after the copy is complete: Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812' Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812' Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.info] Statting file `/1235660859812' Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Uploading `/1235660859812' (flags creat ) Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Closed file `/1235660859812' (handle=ba68d) Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] File `/1235660859812' closed (written: 3.1E, read: 7.0E). Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Setstat file `/1235660859812' (perms) Feb 26 07:14:55 sftp-server[15819]: [ID 702911 local7.notice] Renaming `/1235660859812' to `/consumer_symalllanguages.zip' (Hostnames deleted & directory structure modified for brevity sake) If a transactional based, directory comparison starts at the same time, it will see the transient file as new content, and will prepare to send it as a part of the deployment. However, when the actual transfer phase of the deployment starts, the transient file will have been "renamed", so to OD the transient file no longer exists, and OD fails the job, then rolls back the content. And even though this transient file doesn't exist long, the frequency of this type failure is more common than you'd imagine when hundreds of files are being staged for publishing each day, and multiple jobs are deployed, manually or automatically. I could try to create an inclusion Filter, and am building a list of extensions for that purpose. Any performance issues if this list grows to more than [5|10|20?] different extensions? I could try an exclusion filter, looking for any 13-character all-numeric files; is OD's regex reliable for that? Has anyone else experienced this; and/or can you give any advice on how to prevent it from recurring? Any help is appreciated, KenS

not sure if I am following what your saying, but in each of the OD receivers, you can set a transientDirectory , that is different than your deployment directory.. it will only copy the final files into your real target, and therefore any listeners would only see the files after everything is successfull.

[html]


[/html]

not sure if that helps..