Data deduplication

Data Deduplication essentially refers to the elimination of redundant data. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored. However, indexing of all data is still retained should that data ever be required. Deduplication is able to reduce the required storage capacity since only the unique data is stored. For example, a typical email system might contain 100 instances of the same one megabyte (MB) file attachment. If the email platform is backed up or archived, all 100 instances are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; each subsequent instance is just referenced back to the one saved copy. In this example, a 100 MB storage demand could be reduced to only one MB.

Benefits

How do business benefit from data-deduplication?

• The business benefits from data de-duplication start with increasing overall data integrity and end with reducing overall data protection costs. Data de-duplication lets users reduce the amount of disk they need for backup by 90 percent or more.
• With reduced acquisition costs—and reduced power, space, and cooling requirements—disk becomes suitable for first stage backup and restore and for retention that can easily extend to months.
• With data on disk, restore service levels are higher, media handling errors are reduced, and more recovery points are available on fast recovery media.
• Data deduplication also reduces the data that must be sent across a WAN for remote backups, replication, and disaster recovery.

What all of that really means is that data protection is improved, service is faster, and costs are reduced.

Major Commercial Players

There are plenty of vendors, because data deduplication is a very hot area these days, especially now that the VTL (virtual tape library) vendors are getting involved. There is ExaGrid, Quantum, Avamar (acquired by EMC), Symantec Puredisk, Asigra, Data Domain, Diligent Technologies, Falconstor, and SEPATON. Quantum and NetApp have products in beta.

References

• Biggar, Heidi(2007.12.11). WebCast: The Data Deduplication Effect
• CIMdata (2007-12-27). "Why do Enterprises need Data Deduplication". Press release.
• Using Latent Semantic Indexing for Data Deduplication.
• http://www.eweek.com/c/a/Knowledge-Center/What-Is-the-Difference-Between-Data-Deduplication-File-Deduplication-and-Data-Compression/