FTP component
The FTP component is able to connect to FTP/SFTP servers and read or write files. It can act both as a consumer (read files from the server), and a producer (write files to the server).
Configuration
The FTP component has the following configuration options:
- User Name
- Password
- Private Key File
- Host Name
- Port
- Protocol
- Passive Mode
- Disconnect
- Move
- Read Lock
- Path
- File Name
- Charset
- Auto Create
- File Exist
- Recursive
- Include
- Exclude
- Delete File
- Initial Delay
- Delay
User Name
Description
Username used to log in on the (S)FTP(S) server.
Password
Description
Password used to log in on the (S)FTP(S) server.
Private Key File
Description
Location of a private key to log in on the SFTP server.
Remark
Only applicable to SFTP.
Host Name
Description
URL to the (S)FTP(S) server, e.g. ftp.example.org or IP address, e.g. 149.210.209.32.
Port
Description
Specify the (S)FTP(S) port if the server uses non-default ports. When left empty,
defaults are FTP: 21, SFTP: 22.
Protocol
Options
sftp(default)ftpsftp
Description
Specify whether the server is an FTP server, SFTP server or FTPS Server.
Passive Mode
Options
falsetrue(default)
Description
Specify whether FTP should operate in active or passive mode.
Disconnect
Options
false(default)true
Description
Specifiy whether FTP should force a reconnect on polling.
Move
Description
Relative path to folder where files should be moved to when done processing.
Create and move files into this folder when done processing. This should be a
relative path. In order to move to higher paths use ../[folder name] or
../../[folder name].
Remark
- Files can be moved to folders higher in the hierarchy with
.., for example:../[folder name]or../../[folder name]. - Not applicable when Delete File is set to
true. - The default behavior is to create a directory named
.dovetailand move everything into that directory when files are processed. However some (S)FTP(S) servers don't allow creating.\*files or directories. In this case, you need to change the name of the.dovetaildirectory to something that is accepted by the server.
Read Lock
Options
nonechanged
Description
If set to changed, only consumes the files if it has exclusive read-lock on
the file (i.e. the file is not in-progress or being written). It will wait until
the file lock is granted.
Remarks
- Only applicable for consuming files.
When set to changed:
- It requires write permissions on the (S)FTP(S) server.
- It uses file length or file modification timestamp to detect whether the file is currently being changed or not. It will at least use 1 second to check this and there will also be some time between each check, so consuming files will be slower.
- It will require a lock on the latest file until a new file has been written to the (S)FTP(S) server. It will not be processed until a new file has been dropped at the ftp server.
Path
Description
Path to the target directory. In case of consumer this is the directory where Dovetail will be picking up files from, if used as a producer this is the directory where Dovetail will write files to.
Remark
If you include a / at the end of the path, the ftp component will try to copy
the whole file path and not the file!
File Name
Description
Consume a specific file with this name.
Charset
Description
Specifies the encoding of the file.
Remarks
- Only applicable when producing.
- When processing large text files see Producing large text files below.
Auto Create
Options
false(default)true
Description
If set to true, automatically creates missing directories in the file path.
Remark
Only applicable when producing.
File Exist
Options
Override(default)AppendIgnoreFail
Description
Specify what to do if a file with the same name already exists.
Remark
Only applicable when producing.
Recursive
Options
false(default)true
Description
When consuming, specifies whether Dovetail should process files only in the
specified path (false) or in all subdirectories of the path too (true).
Remark
When set to true you must make sure that your Exclude pattern
will ignore the directory were you place the processed files, otherwise you will
end up with a infinite loop.
Include
Description
Used to include files, if the filename matches the regular expressions pattern.
Remark
Only applicable when consuming.
Exclude
Description
Used to exclude files, if the filename matches the regular expressions pattern.
Remark
Only applicable when consuming.
Delete file
Options
false(default)true
Description
If set to true, the file will be deleted after it is processed.
Remarks
- Only applicable when consuming.
- When set to
true, the Move option is ignored.
Initial Delay
Description
Milliseconds to wait before polling the file/directory starts.
Delay
Description
Milliseconds to wait before the next poll of the file/directory.
Producing large text files (> 10 MB)
When producing large text files like XML or JSON you might want to leave the
charset option empty. This tells Dovetail that it should read the file as binary
which means it does not have to read all the file's content into memory to validate
the charset. This is better for it's performance and can prevent errors parsing
the file later on in the flow.
Consuming folders with a lot of files
When the FTP component acts as a consumer it can't pick up multiple files at the same times and process them. Every time the FTP Component pols the server it will create a list of all the available files and pick one to process. For this reason consuming a folder with a large amount of files can take more time than expected.
To improve the performance you could split up all the files over different subfolders, so you can poll each subfolder with a different FTP component.
Remarks
- Authentication to
SFTPservers can be provided either by specifying a password, or a path to a SSH private key file.- The private key file needs to be of the format
OpenSSH.
- The private key file needs to be of the format
- You can use
${header.<headername>}and${headers.<headername>}variables for all properties of the FTP component