Subscribe

Nov 16, 2016

How to make working with large files less frustrating

  • Connect and support people

    • Do you often find yourself working with large files? Then chances are you’ve encountered hindrances in the form of hard-/software or network limitations more than once. There are some tricks you can use to keep up the pace!

    Back in the early 80’s, when Microsoft developed MS-DOS, according to the myth, Bill Gates said “a 640 KB memory should be enough for anyone”. Today, this is considered one of the biggest misapprehensions in the history of technology. While in those days people mostly worked with small text files, due to enhanced programs – such as CAD, multimedia, and simulation tools – today’s programs are more complicated, expansive, and so are the associated file sizes. The biggest problems with large files of several hundred MB or even reaching GB size can be severe slowdowns of your computer and the matter of transferring them to other devices or storage.

    Summary

    • Make sure your computer is really up to the task
    • Work on large files locally
    • Prefer threading over multiple windows of one program
    • Use exFAT on flash drives and encrypt them, if sent in mail
    • Optimize your network infrastructure
    • Don’t abuse email for file transfer

    Read on for all best practices on how to work productively with large files.

    1. Power Up Your Hardware

    The first thing you want to have a look at is your computer as the foundation of your working speed.

    If you’re working with very large files, especially for CAD/CAM and multimedia type applications, then a 64-Bit machine will be essential to work productively.

    A separate hard drive and not just a swap file or swap partition can also help increase processing speed. Ideally this would be a SSD, with its faster access times and transfer speeds.

    Make sure you have a high-quality video card with plenty of graphic RAM.

    Next you should consider adding more RAM to your computer if there’s space available. A computer can almost always see a speed boost from increasing the amount of RAM in the system.

    But it isn’t just about the RAM. If your CPU is too slow or outdated, then you will not be getting the best out of adding more RAM.

    What about multi-core processors and GPUs? It turns out that there are only very few cases where this hardware comes into play to speed up your device and management of large files.

    2. LAN and WAN Infrastructure Also Plays a Role

    Data transfer rates will still depend upon the speed of the Internet or LAN/WAN links between sender and receiver.

    If you’re working with very large files in a team environment, then it’s important for the hardware and network infrastructure to be capable of handling such large files.

    High network speeds are crucial. This means gigabit Ethernet at the very least. File servers can also be a serious bottleneck. File systems such as NFS, CIFS and Samba may need configuring and adjusting.

    3. Work On Large Files Locally

    Is your internet connection causing you trouble on a regular basis?

    Even though it may take a while to copy a file to your workstation and afterwards back to the server, this practice prevents for internet connectivity issues while working.

    However, be sure to have a procedure in place, when working with team members to ensure for important files not to be overwritten. Measures for this might be versioning, adding change dates or using professional tools, such as CVS, SVN or SCCS.

    4. Don’t Open Multiple Instances of Your Application

    While it may seem comfy to open up all the things you need throughout the day and just leave them open, this is far from best practice.

    Especially when working with large files, e.g. in CAD software, multiple instances of the program with a few gigabytes each can eat up your RAM in no time.

    The consequence being your computer constantly having to swap data between your RAM and hard drive, thus slowing it – and your productivity – down.

    So better close all applications you don’t need at the moment. Furthermore, for most tasks you should prefer threading to opening new instances (this is equivalent to opening new tabs in a browser instead of a new window – although some browsers also open a new process for each tab), sparing your CPU and RAM.

    5. Choosing the Right File System

    Your choice of file system also affects the type of issues you might run into when you want to work with large files.

    • FAT32

    FAT32 is the oldest file system to be considered. Its advantage is that, because of its age, it’s become a standard. You can use it with practically any device that has a USB port, so maximum compatibility is guaranteed.

    On the other hand, a single file to transfer can’t be bigger than 4GB.

    • NTFS

    NTFS is Microsoft’s successor to FAT32, and fixes the 4GB file size limit.

    Its features, e.g. file permissions, changelog or shadow backups, make it ideal for Windows system drives and devices only used with windows.

    Also, in practice there’s no realistic limit on file size anymore.

    The limitations that come with the advantages are compatibility issues with other devices: Macs can only read NTFS devices by default and with Linux it really depends on your luck.

    You can usually convert FAT32 drives to NTFS with few problems (but make sure your data is backed up first to be sure).  Check out Microsoft’s Technet site for instructions on how to convert FAT to NTFS.

    • ExFAT As the Best Choice for Flash Drives

    ExFAT has no file size limit, while being more lightweight than NTFS, abandoning the features that are mainly important if used for the system drive.

    It is also more compatible, working with modern versions of Mac OS X and Linux, too, provided you use an additional program.

    • File Systems outside the Windows Universe

    Linux users are spoilt for choice when it comes to file systems, as along with FAT32 and NTFS, there’s also EXT3, EXT4, XFS, JFS and others.

    Apple Mac OSX uses the Hierarchical File System Plus (HFS+) which provides for an 8 Exabyte maximum file size which should be more than adequate for years to come.

    6. Transfer Large Files Between Devices Efficiently

    Working with large files often involves the need to move the file from one device to another.

    There are a number of ways to achieve this. But not all methods are equal.

    • Email Is a No-Go for File Transfer

    Many people instinctively think of email as their weapon of choice for every kind of file transfer.

    But you should think twice, since email is just not designed for that:

    You will face terrible up- and download speeds, tight restrictions for the size of attachments and the overall usability for file transfer is just not given.

    On top of that, files actually increase in size when added to an email.

    • Using USB Flash Drives to Transfer Large Files

    USB flash drives have a high capacity to store large files, 64 or 128 GB being common storage space nowadays.

    Meanwhile, the copying speed depends on the USB version of course. The one you’re using might be far outdated, the future standard with many advantages being USB-C.

    If you’re planning on sending a flash drive containing confidential data via mail, be sure to encrypt it, e.g. with Veracrypt.

    But there can also be other issues with using USB flash drives with very large files.

    • Using FTP to Transfer Large Files

    To tell the truth, better use SFTP (Secure File Transfer Protocol) or SCP (Secure Copy Protocol) from the start. They encrypt both the login process as well as the data transmission.

    FTP unfortunately is slow, unreliable and insecure, since IDs and passwords are transmitted in clear text across the link and your files can be intercepted.

    Also don’t use the older RCP (Remote Copy Protocol) that is often still found on Unix and Linux systems as it is fundamentally insecure.

    • What About Cloud-Based File Transfer Services?

    Another option is to use a cloud based service such as Dropbox, Box, or Google Drive, or non-account based portals such as WeTransfer.

    These may be a possibility, but data upload and download speeds can be a bottleneck. In part it depends upon the speed of your Internet connection.

    You also may want to consider the issue of data security with cloud service providers.

    • Using TeamViewer File Transfer

    You can use TeamViewer to quickly and easily send large files over the Internet.

    The data transfer is also encrypted, which means it’s secure. It’s also direct i.e. peer-to-peer, and can now run up to 200 MB/s – which is astoundingly fast.

    All you need to transfer your files is an installed full version of TeamViewer.

    Key Takeaways

    So, if you’re a person regularly working with large files, in summary this is what’s important:

    • You should consider a hardware upgrade in terms of enhanced RAM, CPU and potentially GPU.
    • Along with your hardware, your network quality is one of the biggest influencing factors, when talking about file transfer. Arrange for network optimization!
    • However, your best hardware won’t be of any use for long, if you open calculation intensive programs in multiple instances. Better use threading! Also, close all programs you don’t need at the moment.
    • If your internet or local network connection is bad, work on large files locally to prevent disruptions.
    • Think about the best file transfer system considering compatibility and maximum file size requirements.
    • Last, but not least: Don’t get tempted by the imaginative ease of email for file transfer. That’s not their purpose and thus doesn’t work well in practice.