DataCenterNews Asia logo
Specialist data center news for Asia
Story image

Equinix: Creating a distributed data repository at the edge

By Julia Gabel
Thu 16 Nov 2017
FYI, this story is more than a year old

Historically, increasing volumes of large scale datasets - petabytes and above - have been concentrated in centralized corporate data centers which can create data concentration risk.

However, the complexities of these environments and the interactions between data, people, systems, applications, analytics and clouds that lie outside of these data centers can have a detrimental impact on performance and quality.

What’s going on with all this data?

First, data is growing exponentially - from gigabytes to terabytes, to petabytes and now approaching zettabytes. Proportionally more data is being generated each year than what already existed in storage.

Second, data is no longer centralized.

Its creation and processing have shifted to the edge, where distributed data growth and consumption occurs.

This increasing data gravity at the edge is attracting associated applications and services, such as analytics tools drawing real-time insights from multiple, distributed data sources.

The physics of latency, along with ever-increasing bandwidth costs, make backhauling all this data to a centralized data center, far from these applications and services, unsustainable.

Finally, businesses that were once product-driven are becoming increasingly data-driven, as data becomes more valuable to businesses, customers and competitors, and a more desirable target for bad actors.

As more and more data is created, accumulated, updated and accessed at the edge, proper tools must be available that can manage data segmentation logically and geographically to prevent unmanageable data sprawl at the edge.

What’s impeding optimal data access and management at the edge?

A number of obstacles prevent digital businesses from fully leveraging their data capabilities, including:

  • Centralized data architectures address localized data requirements at a medium scale (by today’s standards). They were not designed for large-scale, geographically distributed data access. This demand is well beyond current IT infrastructure and network design limitations.
  • Data sovereignty and privacy regulations force the geographical localization of some data types. These same concerns also make storing data in the cloud as an alternative, a controversial, if not impossible, option.
  • Corporate policy and industry compliance requirements can also limit what can be done with data.
  • As businesses and applications struggle to shift to the edge, the data management and security capabilities needed to control access are fragmented or worse, absent.
  • New data is continuously being generated at the edge, either creating more sprawl, or experiencing a bandwidth/latency backhaul problem with ongoing capacity and QoE issues and inherent risks.

There has to be a better solution

In our last article in this series, Data is Pouring Over the Edge, we described using a digital edge node, as part of an Interconnection Oriented Architecture (IOA) strategy, to act as an interconnection hub, tailored for local or shared data services at specific geographic locations.

An IOA framework leverages distributed digital edge nodes to create distributed data repositories that can be controlled, improving performance and security in multiple edge locations, while optimizing wide area networking.

Using distributed storage platforms that apply erasure coding to deploy a single namespace data service (optimized for high availability and data protection) in all edge node locations, you can make your data immutable, essentially shielding it against human error and data corruption, and dramatically increase availability.

The data service can use policy-based controls to address logical and geographical data segmentation, and can support cloud provider’s large-scale, multi-tenant data services (i.e., private cloud storage, object storage, etc.).

You can geographically place data nodes (Data Hubs) in each edge node, as well as in cloud environments.

Built-in algorithms interpret policies and store the actual data in a way that protects it from device, location or even regional failures and breaches, without losing data or access capabilities.

This strategy offers far more protection than a data “copy” approach and uses much less storage. Data services can also be optimized for integration, supporting multiple interfaces (e.g., web, APIs, file system) as part of the data abstraction layer across multiple underlying technologies realized with the IOA framework (see diagram below).

A Distributed Data Repository Infrastructure with Global Namespace

To create a distributed data repository at the edge, follow these steps:

  1. Design the geographic placement of data nodes. Even though implementations can vary, the best practice is to have a minimum of four locations (e.g., cloud environment, number of virtual machines). For optimal protection, deploy 16 nodes (four locations each with four nodes).
  2. Place the first data node at an edge location and start the namespace data service. As more data nodes are added, your private data cloud/namespace will expand in size and the service will update itself.
  3. Integrate the service with boundary control, inspection zone(s), policy enforcement and application API management.
  4. Apply event processing and monitoring.
  5. Establish logical capacity buckets and access groups with protection and placement policies.
  6. Continue to add more data nodes to scale namespace capacity. Data nodes deployed in the cloud can be backed by cloud-provided storage (e.g., AWS S3), and control policies will determine cloud use.

The benefits of a distributed data repository at the edge

A distributed data service with centralized management solves the following constraints:

  • It’s designed to handle zettabyte-scale data sizes.
  • It’s data accessibility significantly reduces the volume of data moved and copied over long distances and centrally stored, which reduces WAN bandwidth and storage costs.

It also enables the following benefits:

  • All data can be automatically encrypted at the data layer and identity keys can also be dispersed (not stored in any single place).
  • Reduction of WAN bandwidth and storage costs
  • Greater security and policy management integration ensures compliance consistency in all regions.
  • Any non-latency-sensitive data requirements can be addressed using the data repository as primary storage and/or an archive. An eventual consistency model can also be applied to achieve high availability that informally guarantees that, if no new updates are made to a given data item, all accesses to that item will eventually return the last updated value.

Examples of a distributed data repository’s use include: shared drive, package distribution for applications/containers, logging repository, staging area for analytics, etc.

Article by Olu Rowaiye, Equinix blog network 

Related stories
Top stories
Story image
Tech job moves - Forcepoint, Malwarebytes, SolarWinds & VMware
We round up all job appointments from May 13-20, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Intel unveils new investments for data center sustainability
Intel has announced two new investments, continuing its efforts to create more sustainable data center technology.
Story image
Digital Transformation
The Huawei APAC conference kicks off with digital transformation
More than 1500 people from across APAC have gathered for the Huawei APAC Digital Innovation Congress to explore the future of digital innovation.
Story image
Legrand unveils Nexpand, a data center cabinet platform
Legrand has unveiled a new data center cabinet platform, Nexpand, to offer the necessary scalability and future-proof architecture for digital transformation.
Story image
Digital Edge chooses Nortek’s StatePoint for new data center
Digital Edge will use Nortek's StatePoint liquid cooling technology in its new data center, the first commercial colocation operator in Asia to do so.
Story image
Energy storage demand momentum continues, says BYD
BYD has announced an expansion of its production capacities and will deliver 250,000 units of its energy storage system, BYD Battery-Box Premium.
Story image
Data and analytics could be key to higher selling prices in APAC
Sisense's latest report has found that almost half of data professionals in APAC think customised data and analytics can create better selling prices for their products.
Story image
SAS Viya on Microsoft Azure to deliver 204% return - study
The Forrester Total Economic Impact study finds SAS Viya on Microsoft Azure brings a 204% return on investment over three years.
Story image
Power / Energy
DigitalBridge makes $30 million equity investment in LEDC
Leading Edge Data Centres (LEDC) has announced it has secured an AUD$30 million equity investment in its regional edge network from an affiliate of DigitalBridge Group, DigitalBridge.
Story image
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Story image
Data Center
Digital Edge to build South Korea's largest commercial data center
The project will be the largest commercial data center project in South Korea with total IT power of 120MW and a capital investment of more than KWR$1 trillion.
Story image
The 'A-B-C' of effective application security
Software applications have been a key tool for businesses for decades, but the way they are designed and operated has changed during the past few years.
Story image
Video: 10 Minute IT Jams - SoftIron CMO on Data Center Sustainability
In a special Power/Energy feature week presentation, we are joined by SoftIron CMO Andrew Moloney.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Digital Transformation
Multiplex, NEXTDC making strong progress on S3 data centre
Multiplex has made a significant achievement on Stage 1 of NEXTDC’s S3 data centre, ‘topping out’ the structure in the Artarmon on Sydney’s lower North Shore.
Story image
Colt connectivity with AWS increases services in Asia
Colt Technology Services expands cloud connectivity to AWS Direct Connect Hosted services, with speeds of up to 10 Gbps in Asia.
Story image
ABB unlocks sustainable energy upgrades for data centers
ABB says its new microgrid solutions will get data centers ready for the green revolution and use their own energy sources with a reduced carbon footprint.
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Cisco reveals new tech, intends to prevent network issues
Cisco has revealed new technology intended to mitigate costly disruptions by aiding IT teams in learning, predicting and planning.
Story image
Orange moves Siemens AG’s entire operations to a SD-WAN
Orange Business Services has migrated Siemens AG's entire global operations, 1168 sites across 94 countries, to a SD-WAN
Story image
Data Center
CBRE finds record levels of investment in APAC data centers
CBRE's new report finds direct investment in the sector more than doubled in 2021, surpassing investment volumes for the past four years combined
Story image
Digital Transformation
EdgeConneX enters Indonesia, plans for data center campus
EdgeConnex has announced it is expanding its presence in Asia with the acquisition of GTN Data Center in Indonesia.
Story image
Daikin and SP Group to build new energy efficient district cooling system
The project, set to be complete by 2025, will create a system with a cooling capacity of up to 36,000 refrigerant tonnes (RT). 
Story image
Why tenant consolidation is critical to cloud success
Consolidating tenants can improve cost management, security and engagement after a flurry of reactive activity following the widespread shift to remote operations.
Story image
Akamai announces new products across security, computing
Akamai has announced a series of new products and updates to existing products across its security and compute product lines, including its entry into the infrastructure as a service (IaaS) market.
Story image
Telstra expands business offerings in the Philippines
The expansion aims to offer more choice for customers and enhance connectivity into the Philippines, and within the country.
Story image
Power / Energy
Keysight Technologies introduces new next-gen DPT solution
Keysight Technologies has announced its new next-generation Double-Pulse Tester (DPT) with the PD1550A Advanced Dynamic Power Device Analyser.
Story image
Tech Data
Tech Data to use Pluribus Networks’ cloud solutions in APAC
Tech Data says using Pluribus Networks' Unified Cloud Fabric solution will be a "game-changer" for its data center infrastructure customers and partners.
Story image
NTT launches IoT Services for Sustainability offering
"We know what actions are needed to build a more sustainable future and have a robust suite of technologies available to help deliver this impact."
Story image
New strategies for cloud-native attacks - Aqua Security
New research from Aqua Security reveals attackers are using more sophisticated techniques to target cloud-native environments.
Story image
Siemens showcases new automated solutions for data centers
Siemens has implemented new automated solutions and AI in the Baltic region's largest data center, providing insight into the future of data center management.
Story image
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.
Story image
Databricks grows in APAC market, expands into Korea
Databricks officially launches a local office in Seoul, Korea, building on existing partnerships with Cloocus, Megazone and the Weverse Company
Story image
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.
Story image
Genetec launches new enclosure management system for data centers
Genetec has released a new enclosure management solution that will give data centers the ability to secure, monitor and manage access to racks and cabinets remotely.
Story image
Red Hat
Red Hat expands capabilities to provide streamlined application development in cloud
"Application development is undergoing significant change and developers need tools to support this transformation."
Story image
SpaceDC partners with Aofei for data center sales in Asia
SpaceDC has partnered with Aofei Data International to sell Aofei's data centers, CDN and SDN in China.
Story image
Talend introduces new data health solutions for businesses
Talend has announced its latest version of Talend Data Fabric, with the release of Talend Trust Score enabling data teams to establish a foundation for data health.
Story image
Report - Data investment the key to better business growth
New research from Digital Realty has revealed that almost half (47%) of IT leaders globally believe their business investment in data systems and infrastructure is a key obstacle or concern.
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
AyalaLand and FLOW partner for data center development
AyalaLand Logistics Holdings Corp (ALLHC) and FLOW Digital Infrastructure have entered into a framework agreement to bolster the development of carrier-neutral data centers in the Philippines.
Story image
RDA and MVGX partner for sustainable data center development
Red Dot Analytics (RDA) and MetaVerse Green Exchange (MVGX) have entered a strategic partnership to make Singapore's data center development and operations more sustainable.
Story image
New high-performance cable in the works for Asia
A new high-performance submarine cable is being built to enhance connectivity between Hong Kong, China and Southeast Asia.
Story image
AirTrunk boosts Japan presence with West Tokyo data center
AirTrunk is planning to build TOK2, a new hyperscale data center in Japan which will strengthen the company’s presence in the country.