Linux Containers Authors: Flint Brenton, Liz McMillan, John Katrick, Gordon Haff, Pat Romanski

Related Topics: Linux Containers

Linux Containers: Article

Alternative Ways to Download Files with Linux

A sneak peek

Fedora Core (http://fedora.redhat.com) is one of the world's most popular Linux distributions in part from support by Red Hat and a strong community of users. It also has been a proving ground for Red Hat to develop technologies that will eventually make their way into Red Hat Enterprise Linux.

Recently Red Hat has spun off the Fedora Core Project into its own organization in much the same way that the Mozilla project was spun out of Netscape back in the late 1990s. Both Fedora and Red Hat Enterprise Linux have proven to be raging successes in their own right for both Red Hat and Linux users who choose to pursue a do-it-yourself approach to Linux. For those of you looking for additional resources to learn more about these distributions, you could consider the Red Hat Fedora and Enterprise Linux 4 Bible written by Christopher Negaus.

Christopher Negus has been working with UNIX systems, the Internet, and (most recently) Linux systems for more than two decades at AT&T Bell Laboratories, UNIX System Laboratories, and Novell, helping to develop the UNIX operating system. Features from many of the UNIX projects Chris worked on at AT&T have found their way into Red Hat Enterprise Linux, Fedora, and other Linux systems. Chris has written several books on UNIX and the Internet; his latest book is the Red Hat Fedora and Enterprise Linux 4 Bible (excerpted below) and released in July 2005. It discusses the features and capabilities of both operating systems. His book is written as a resource for users trying to adopt either of these technologies.

With your Fedora Core system connected to the Internet, you can take advantage of dozens of tools for browsing the Web, downloading files, getting e-mail, and communicating live with your friends. In most cases, you have several choices of GUI and command-line applications for using Internet services from your Linux desktop or shell. However, there are some utilities you may not be familiar with like wget, scp, and sftp.

Excerpt from Chapter 9...

Getting Files with wget
If you already know where a file is on the network, there are more efficient ways of downloading that file than opening an FTP session, moving around the FTP server, and running the get command. The wget command is a simple, efficient tool for doing non-interactive downloads of files over the Internet.

If there is a file you want to download from an FTP site or Web server (HTTP), and you know exactly where the file is, wget is a good way to download. The wget command is very useful if you want to copy a whole site, recursively, from one computer to another (for example, containing user home directories). When downloading from FTP sites, wget can let you just download as the anonymous user or add your own user name and password to the command line.

Downloading a File with User Name and Password
If you are doing an FTP file copy and need to log in as a user other than anonymous, you can add that information to the command line or to a .netrc file in your home directory (type man netrc to see the format of that file). Here is an example of adding the password to the command line:

$ wget ftp://joe:[email protected]/memol.doc .

CAUTION: Adding a password to a command line leaves the password exposed to onlookers. This practice is generally discouraged, except in cases where no one can see your monitor or your history files.

In the previous example, the user logs in as joe with the password my67chevy. The wget then copies the file memo1.doc from the current directory on the host computer named ftp.handsonhistory.com. That current directory is most likely /home/joe.

Downloading a Whole Web Site
Using wget, you can download a large number of files from Web servers as well. The wget command downloads files using the http protocol, if file addresses begin with http://. Downloading a single file, you would use the same form as you would for an FTP file (for example: wget http://host/file.). The best wget option for HTTP downloads is -r (recursive).

A recursive download lets you choose a point at a Web site and download all content below that point. Here is an example of a recursive download used to download the contents of the www.example.com Web site.

$ wget -r http://www.example.com

In this example, the HTML pages, images, and other content on the www.example.com Web site are copied below the current directory in the directory www.example.com. This is useful if you want to gather the contents of a Web site but don't have login access to that site. Because content is taken by following links, if there is content in a directory at the Web site that isn't in a link, it won't be downloaded.

Downloading an entire Web site can result in a massive amount of data being downloaded. If you want only part of a Web site, start from a point lower in the site's structure. Or, as an alternative, you can limit the number levels the wget will go down the site structure. Using the -l option (l as in level), the following example gets two levels of HTML content:

$ wget -r -l 2 http://www.example.com

To mirror a site, you can use the -m option instead of -r. With wget -m http://site, it is like asking to download an infinite number of levels recursively (-r -l inf), keep current time stamps (-N), and keep FTP directory listings (-nr).

Continuing a Download
In the old days, if you were downloading a particularly large file (such as an ISO image of a CD or DVD), if the download stopped for some reason (a disconnected network or errant reboot), you needed to start all over. With wget, you can choose to restart a download and have it continue right where it left off. This has been a lifesaver for me on many occasions.

Let's say that you were downloading a 4GB DVD ISO image named mydvd.iso from the site ftp://ftp.example.com and you killed the wget process by mistake after about 3GB of download. Make sure that your current directory is the one that contains the partially downloaded ISO. Then run the same wget command you did originally, adding the -c (continue) option as follows:

$ wget -c ftp://ftp.example.com/mydvd.iso

If you had not used the -c option, in this case, a new download would have started using the file name mydvd.iso.1 to download to.

NOTE: Another command that you might be interested in, similar to wget, is the curl command. Like wget, curl can download files using the ftp or http protocols. Curl can also do multiple file transfers on the same connection.

# ssh [email protected] "tail -f /var/log/messages"
[email protected]'s password:

After you typed the password in the preceding case, the last several lines of the /var/log/messages file on the remote computer would be displayed. As messages were received, they would continue to be displayed until you decided to exit (press Ctrl+C to exit the tail command).

NOTE: Find out more about the ssh command from the SSH Web site (www.openssh.org).

Using scp for Remote File Copy
The scp command is a simple yet secure way of copying files among Linux systems. It uses the underlying ssh facility, so if ssh is enabled, so is scp. Here is an example of using scp to copy a file from one computer to another:

# scp myfile toys.linuxtoys.net:/home/chris
[email protected]'s password: ******

In this example, the file myfile is copied to the computer named toys.linuxtoys.net in the /home/chris directory. If you don't provide a user name (as is the case here), scp assumes you are using the current user name. Unlike some tools that provide remote login, scp and ssh do allow you to login as root user over the network, by default. (Many people turn off this feature for security reasons.)

To use scp with a different user name, you can append the user name with an @ character. For example, [email protected]:/home/chris would attempt to log in as the user named chris to do the file copy.

The first time you connect to a remote computer using scp or ssh, those commands try to establish the authenticity of the remote host. If it cannot establish the host's authenticity, it will display the RSA key fingerprint and ask you if you want to continue. If you type yes, scp will not question the authenticity of that computer again for subsequent scp commands.

However, if the RSA key fingerprint should change in the future for the remote computer (which will happen if, for example, the operating system is reinstalled on that computer), scp will refuse to let you connect to that remote computer. To override that refusal, you need to edit your $HOME/.ssh/known_hosts file and delete the entry for the remote computer. You can then verify the authenticity of the remote computer and continue to use scp.

The sftp command, which also communicates using secure ssh protocols, is a command for copying files from an FTP server. It is considered a more secure way of getting files from an FTP server that has an sshd server running.

.  .  . 

You can receive more helpful tips on using Fedora and Red Hat Enterprise Linux 4 in Chris's book, Red Hat Fedora and Enterprise Linux 4 Bible available at your local retailer.

More Stories By Mark R. Hinkle

Mark Hinkle is the Senior Director, Open Soure Solutions at Citrix. He also is along-time open source expert and advocate. He is a co-founder of both the Open Source Management Consortium and the Desktop Linux Consortium. He has served as Editor-in-Chief for both LinuxWorld Magazine and Enterprise Open Source Magazine. Hinkle is also the author of the book, "Windows to Linux Business Desktop Migration" (Thomson, 2006). His blog on open source, technology, and new media can be found at http://www.socializedsoftware.com.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@ThingsExpo Stories
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect at GE, and Ibrahim Gokcen, who leads GE's advanced IoT analytics, focused on the Internet of Things / Industrial Internet and how to make it operational for business end-users. Learn about the challenges posed by machine and sensor data and how to marry it with enterprise data. They also discussed the tips and tricks to provide the Industrial Internet as an end-user consumable service using Big Data Analytics and Industrial C...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
With privacy often voiced as the primary concern when using cloud based services, SyncriBox was designed to ensure that the software remains completely under the customer's control. Having both the source and destination files remain under the user?s control, there are no privacy or security issues. Since files are synchronized using Syncrify Server, no third party ever sees these files.
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Internet-of-Things discussions can end up either going down the consumer gadget rabbit hole or focused on the sort of data logging that industrial manufacturers have been doing forever. However, in fact, companies today are already using IoT data both to optimize their operational technology and to improve the experience of customer interactions in novel ways. In his session at @ThingsExpo, Gordon Haff, Red Hat Technology Evangelist, shared examples from a wide range of industries – including en...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Rodrigo Coutinho is part of OutSystems' founders' team and currently the Head of Product Design. He provides a cross-functional role where he supports Product Management in defining the positioning and direction of the Agile Platform, while at the same time promoting model-based development and new techniques to deliver applications in the cloud.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
delaPlex is a global technology and software development solutions and consulting provider, deeply committed to helping companies drive growth, revenue and marketplace value. Since 2008, delaPlex's objective has been to be a trusted advisor to its clients. By redefining the outsourcing industry's business model, the innovative delaPlex Agile Business Framework brings an unmatched alliance of industry experts, across industries and functional skillsets, to clients anywhere around the world.
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...