Singapore's Digital Archives Are Drowning in Duplicate Images — The Numbers Tell a Stark Story

Singapore's public and private sector organisations are sitting on hundreds of millions of redundant digital image files. According to data published by the Infocomm Media Development Authority in its 2025 Digital Infrastructure Report, government agencies collectively managed more than 4.8 petabytes of unstructured media data as of December 2025 — a figure that specialists in the field say carries a duplication rate typical of large institutional archives, which industry benchmarks place at anywhere between 25 and 40 percent of total stored content.

The timing matters. Singapore is midway through the Smart Nation 2.0 initiative, which commits ministries and statutory boards to migrating legacy document stores into AI-ready formats by the end of 2027. Duplicate image replacement — the process of identifying, tagging and substituting redundant visual files with verified canonical versions — sits at the bottleneck of that pipeline. Every duplicated image that enters a training dataset skews model outputs; every duplicated file stored in a government cloud bucket costs real money on a per-gigabyte basis.

Where the Problem Shows Up Most

The National Library Board's NewspaperSG archive, accessible from its Victoria Street headquarters, digitised roughly 1.2 million newspaper pages between 2019 and 2024. Librarians working on the project have noted publicly that scanning batches frequently produce near-duplicate images — slightly different exposure levels, minor rotation offsets — that automated optical character recognition systems treat as entirely separate files. The Housing and Development Board faces a parallel issue: its estate management portal, which serves more than one million flat owners across towns from Woodlands to Tampines, stores images of defect reports, renovation permits and inspection logs. HDB's own public tender documents from 2024 referenced plans to implement deduplication tooling as part of a broader S$34 million digital facilities management upgrade.

The Monetary Authority of Singapore added another dimension when it updated its Technology Risk Management Guidelines in January 2024. Financial institutions under MAS oversight are required to maintain audit-grade image records of physical documents — loan applications, know-your-customer identity scans — with version integrity controls. Banks operating out of the Marina Bay Financial Centre have told industry forums that image deduplication is now a compliance question, not merely a housekeeping one, because duplicate files can obscure audit trails.

What Deduplication Actually Costs — and Saves

Storage is not cheap in Singapore's data centre market. The JTC Corporation's industrial estates at Jurong and Tuas host several hyperscale and colocation facilities; published rack rates from operators in those corridors averaged between S$180 and S$240 per terabyte per year for enterprise-grade storage in 2025. If a mid-sized government statutory board carries 50 terabytes of image data with a 30 percent duplication rate, that is roughly 15 terabytes of redundant files costing the public purse an estimated S$2,700 to S$3,600 annually — a modest line item individually, but multiplied across 16 major ministries and more than 60 statutory boards, the aggregate runs into seven figures.

Software vendors pitching deduplication solutions to Singapore's public sector have cited reduction ratios of 3:1 to 5:1 for document-image repositories, meaning an organisation with one petabyte of image storage could theoretically reduce active storage to between 200 and 333 terabytes. The Government Technology Agency, based at Sandcrawler Building in one-north, has been evaluating hash-based and perceptual-hashing deduplication frameworks as part of its data management playbook for GovTech's suite of shared services platforms.

For businesses and agencies still in planning mode, the practical path forward involves three steps: a baseline audit using perceptual hash tools such as open-source libraries like ImageHash to generate a duplication rate estimate; a tiered deletion policy that archives rather than permanently removes files for a defined retention window; and integration of deduplication checks at the point of ingest rather than as a retrospective cleanup. The cost of ingest-level deduplication is consistently lower than remediation after the fact. With Singapore's AI governance framework requiring clean, auditable training data for any model deployed in public-facing services, the organisations that sort this out before the 2027 Smart Nation deadline will be better placed than those that do not.