DataCenterNews Asia Pacific logo
Specialist data center news for Asia Pacific
Story image

Equinix: Creating a distributed data repository at the edge

FYI, this story is more than a year old

Historically, increasing volumes of large scale datasets - petabytes and above - have been concentrated in centralized corporate data centers which can create data concentration risk.

However, the complexities of these environments and the interactions between data, people, systems, applications, analytics and clouds that lie outside of these data centers can have a detrimental impact on performance and quality.

What's going on with all this data?

First, data is growing exponentially - from gigabytes to terabytes, to petabytes and now approaching zettabytes. Proportionally more data is being generated each year than what already existed in storage.

Second, data is no longer centralized.

Its creation and processing have shifted to the edge, where distributed data growth and consumption occurs.

This increasing data gravity at the edge is attracting associated applications and services, such as analytics tools drawing real-time insights from multiple, distributed data sources.

The physics of latency, along with ever-increasing bandwidth costs, make backhauling all this data to a centralized data center, far from these applications and services, unsustainable.

Finally, businesses that were once product-driven are becoming increasingly data-driven, as data becomes more valuable to businesses, customers and competitors, and a more desirable target for bad actors.

As more and more data is created, accumulated, updated and accessed at the edge, proper tools must be available that can manage data segmentation logically and geographically to prevent unmanageable data sprawl at the edge.

What's impeding optimal data access and management at the edge?

A number of obstacles prevent digital businesses from fully leveraging their data capabilities, including:

  • Centralized data architectures address localized data requirements at a medium scale (by today's standards). They were not designed for large-scale, geographically distributed data access. This demand is well beyond current IT infrastructure and network design limitations.
  • Data sovereignty and privacy regulations force the geographical localization of some data types. These same concerns also make storing data in the cloud as an alternative, a controversial, if not impossible, option.
  • Corporate policy and industry compliance requirements can also limit what can be done with data.
  • As businesses and applications struggle to shift to the edge, the data management and security capabilities needed to control access are fragmented or worse, absent.
  • New data is continuously being generated at the edge, either creating more sprawl, or experiencing a bandwidth/latency backhaul problem with ongoing capacity and QoE issues and inherent risks.

There has to be a better solution

In our last article in this series, Data is Pouring Over the Edge, we described using a digital edge node, as part of an Interconnection Oriented Architecture (IOA) strategy, to act as an interconnection hub, tailored for local or shared data services at specific geographic locations.

An IOA framework leverages distributed digital edge nodes to create distributed data repositories that can be controlled, improving performance and security in multiple edge locations, while optimizing wide area networking.

Using distributed storage platforms that apply erasure coding to deploy a single namespace data service (optimized for high availability and data protection) in all edge node locations, you can make your data immutable, essentially shielding it against human error and data corruption, and dramatically increase availability.

The data service can use policy-based controls to address logical and geographical data segmentation, and can support cloud provider's large-scale, multi-tenant data services (i.e., private cloud storage, object storage, etc.).

You can geographically place data nodes (Data Hubs) in each edge node, as well as in cloud environments.

Built-in algorithms interpret policies and store the actual data in a way that protects it from device, location or even regional failures and breaches, without losing data or access capabilities.

This strategy offers far more protection than a data “copy” approach and uses much less storage. Data services can also be optimized for integration, supporting multiple interfaces (e.g., web, APIs, file system) as part of the data abstraction layer across multiple underlying technologies realized with the IOA framework (see diagram below).

A Distributed Data Repository Infrastructure with Global Namespace

To create a distributed data repository at the edge, follow these steps:

  1. Design the geographic placement of data nodes. Even though implementations can vary, the best practice is to have a minimum of four locations (e.g., cloud environment, number of virtual machines). For optimal protection, deploy 16 nodes (four locations each with four nodes).
  2. Place the first data node at an edge location and start the namespace data service. As more data nodes are added, your private data cloud/namespace will expand in size and the service will update itself.
  3. Integrate the service with boundary control, inspection zone(s), policy enforcement and application API management.
  4. Apply event processing and monitoring.
  5. Establish logical capacity buckets and access groups with protection and placement policies.
  6. Continue to add more data nodes to scale namespace capacity. Data nodes deployed in the cloud can be backed by cloud-provided storage (e.g., AWS S3), and control policies will determine cloud use.

The benefits of a distributed data repository at the edge

A distributed data service with centralized management solves the following constraints:

  • It's designed to handle zettabyte-scale data sizes.
  • It's data accessibility significantly reduces the volume of data moved and copied over long distances and centrally stored, which reduces WAN bandwidth and storage costs.

It also enables the following benefits:

  • All data can be automatically encrypted at the data layer and identity keys can also be dispersed (not stored in any single place).
  • Reduction of WAN bandwidth and storage costs
  • Greater security and policy management integration ensures compliance consistency in all regions.
  • Any non-latency-sensitive data requirements can be addressed using the data repository as primary storage and/or an archive. An eventual consistency model can also be applied to achieve high availability that informally guarantees that, if no new updates are made to a given data item, all accesses to that item will eventually return the last updated value.

Examples of a distributed data repository's use include: shared drive, package distribution for applications/containers, logging repository, staging area for analytics, etc.

Article by Olu Rowaiye, Equinix blog network 

Related stories
Top stories
Story image
Network Management
Data is growing at breakneck speed, but are we optimising its value?
Data lies at the heart of digital transformation, as every digital touchpoint translates to a data point. In this digital-first world, data is being created everywhere today – at breakneck speeds.
Story image
Data Protection
Cloudflare brings Data Localisation Suite to more APAC businesses
This allows any business in these countries to service their data locally while benefiting from the speed, security, and scalability of Cloudflare’s global network.
Story image
Data Centre Cooling
The world is heating up, but data centres should keep their cool
With the world heating up, the challenge of keeping data centres cool becomes more complex, expensive and power intensive.
Story image
Artificial Intelligence
ASUS Servers announce AI developments at NVIDIA GTC
The Taiwanese multinational now offers NVIDIA-certified servers with H100 Tensor Core GPU and AI enterprise software suite.
Story image
Sustainable IT
Equinix partners NUS to use hydrogen tech in data centres
The partners will develop hydrogen fuel technologies for green data centres in tropical climates, and for use in Equinix’s global network.
Story image
IT Automation
Juniper Networks announces expansion of Apstra Software with Apstra Freeform
The newly announced Apstra Freeform technology will give customers the ability to manage and automate operations for data centers regardless of the architecture.
Story image
Honeywell launches Data Center Suite for business outcomes
Honeywell has launched its Data Center Suite, a portfolio of outcome-based software offerings to help data centre managers and owners.
Story image
SoftIron announces its newest flagship offering, HyperCloud
SoftIron has announced HyperCloud, the world's first full turnkey, completely integrated and supported Intelligent Cloud Fabric and the company's newest flagship offering.
Story image
Worldwide 5G mobile data traffic exploding - report
"With 5G, there is a wider range of deployment scenarios, forcing vendors to provide comprehensive solutions to support every need."
Story image
Growth in hyperscale data centres to increase shortage of IT workers
New Zealand's tech worker capacity is set to come under increasing pressure as the number of hyperscale data centres grows.
Story image
Equinix invests $23m to expand ME2 data centre in Melbourne
Equinix has completed the second phase expansion of its ME2 International Business Exchange data centre, located in Port Melbourne.
Story image
Fortinet unveils compact firewall for hyperscale data centres, 5G networks
"Fortinet’s dedication to pushing the boundaries of what is possible in security performance has yielded the most powerful compact firewall yet."
Story image
Optical Networking
NEC predicts AON as a next-generation infrastructure
NEC's open optical transmission devices support multi-vendor configurations, allowing customers to procure and combine equipment from multiple vendors.
Story image
Zetaris is changing the way we think about data virtualisation
Zetaris was launched on the Microsoft Marketplace and Ingram Micro Cloud Marketplace in Australia in 2020 and has since expanded into nine global markets.
Story image
Data analytics
Srisawan Hospital to enhance patient experience with InterSystems TrakCare
The new Srisawan Hospital in Bangkok has chosen InterSystems TrakCare to help create enhanced patient experiences and promote further digital engagement.
Story image
Talend announces support for Amazon Redshift Serverless
Talend has announced its support for Amazon Redshift Serverless, with the company saying the integration reinforces its commitment and leadership in supporting businesses.
Aws Marketplace
Learn how to implement a backup and recovery plan for a new generation of Kubernetes-based modern applications
Link image
Story image
Sustainable IT
Empyrion DC announces 40MW green data center in South Korea
Empyrion DC has announced it is developing a 40MW green data center in Gangnam, Seoul, South Korea (GDC).
Story image
InterSystems releases updates to its IRIS data platform
Provider of next-generation solutions InterSystems has announced a series of new releases to its award-winning InterSystems IRIS data platform.
Story image
Digital Transformation
Nanyang Technological University Singapore builds digital brand presence
Leveraging the customisation features of Sitefinity DX, non-technical users could upload content and create design pages and boost work productivity. 
Story image
Seagate announces next gen advanced storage arrays
The new Exos X systems feature up to twice the performance of the previous generation and enhanced enterprise-class durability, the company states.
Story image
Edge Computing
NTT launches Edge-as-a-Service to accelerate automation
"Minimum latency, maximum processing power, and global coverage are exactly what enterprises need to accelerate their digital transformation journeys.”
Story image
DCI plans to build new cloud edge data centre in Canberra
DCI is one of the first to commit to the Precinct which has a focus on defence, space, cybersecurity and high-tech manufacturing sectors.
Story image
NCS, FPT Software launch Strategic Delivery Centre in Vietnam
The new partnership is designed to support increasing demand for high quality digital services across the region.
Story image
Data Protection
iseek secures Queensland Government data centre contract
iseek secures the Queensland Government's core network data centre as-a-service contract after a competitive procurement process undertaken by the CITEC.
Story image
Data center
Australia’s data centre pioneer still leading after 22 years
We look at the fascinating success of Macquarie data centre's over its 22 year life span and how they continue to innovate in a highly contested sector.
Story image
Google Cloud Platform
Google Cloud to open first cloud region in NZ - among others
Google Cloud has announced plans to bring three new cloud regions, one each in New Zealand, Malaysia and Thailand.
Story image
Stellar financial result after major strategic moves by Superloop
We get a glimpse under the hood at the financial results from 2022 for the connectivity giant Superloop.
Story image
ManageEngine unveils SaaS availability of Analytics Plus
ManageEngine's Analytics Plus is now available as a software as a service (SaaS) offering, enabling users to set up a completely functional and integrated analytics platform anywhere in under a minute.
AWS Marketplace
Whitepaper: A practical guide for mitigating risk in today’s modern applications
Link image
Story image
Machine learning
Oracle announces MySQL HeatWave for Amazon Web Services
MySQL HeatWave is a service that combines OLTP, analytics, machine learning, and machine learning-based automation. 
Story image
VMware advances multi-cloud management with VMware Aria
Managing apps and infrastructure in a multi-cloud, especially public cloud, and multi-technology environment is complex.
Story image
Data Centre Maintenance / Management
Schneider Electric backs new Leading Edge data centre in Australia
As a result of the new project, regional Australian businesses and communities will likely have greater access to distributed cloud networks.
Story image
DCI Data Centers breaks ground on AKL02 center
DCI Data Centers has commenced construction on Auckland's largest data center.
Story image
Software Defined Wide Area Network
Axiata, Versa Networks partner for enterprise SASE in Asia
Axiata has partnered with Versa Networks to deliver Secure Access Service Edge (SASE) technology to rapidly digitalising Asian enterprises.
Story image
Cloudera launches all-in-one data lakehouse cloud service
CDP One makes it faster, easier and less risky for businesses to move to the cloud and migrate existing workloads to a modern data architecture.
Story image
VMware extends collaboration with Microsoft for enterprise workloads in Azure
Mutual customers will have the choice to purchase Azure VMware Solution through the VMware Cloud Universal program.
Story image
SnapLogic named Visionary in two Magic Quadrant categories
SnapLogic has announced that it is the only iPaaS (Integrated Platform as a Service) vendor to be named a Visionary in two Magic Quadrant categories.
Story image
Sustainable IT
New report calls for tighter guidelines on data centre sustainability
A new Cushman & Wakefield report is calling for water consumption and carbon emissions to be measured in addition to power usage.
Story image
Data center
Macquarie Asset Management acquires stake in ST Telemedias VIRTUS Data Centres
"We will further strengthen VIRTUS' focus on sustainability by backing investment in its technology and enhancing the lifecycle management of its equipment."
Story image
Digital Transformation
NTT launches its Cyberjaya 6 data center in Malaysia
NTT expands its hyperscaler footprint in Malaysia with its sixth data center facility, supporting the growing digital economy.
Story image
Network Infrastructure
Vertiv launches solutions to better manage edge computing
Vertiv has introduced new power and cooling solutions for the edge of the network, including the addition of lithium-ion models to a leading on-line UPS family.