Singapore's Duplicate Image Problem: The Hidden Numbers Driving a Digital Clean-Up

Singapore's public and private sector databases collectively harbour hundreds of millions of duplicate images, a problem that has quietly ballooned alongside the Republic's push to digitise everything from HDB flat inspections to hawker centre licensing. The numbers behind the duplication crisis are only now coming into sharper focus as organisations begin auditing their storage estates ahead of new data governance benchmarks set for review in the third quarter of 2026.

The timing matters because Singapore's Infocomm Media Development Authority has been pressing agencies and enterprises to meet updated data quality standards under the broader Digital Connectivity Blueprint, which outlines infrastructure priorities through 2030. Bloated image repositories sit squarely in the crosshairs of that agenda. Redundant files inflate cloud storage bills, slow retrieval systems, and — critically — introduce compliance risk when multiple versions of the same identity document or property photograph exist without a clear record of which copy is authoritative.

What the Numbers Actually Show

Industry estimates, which individual vendors and consultancies have cited in separate briefings, suggest that duplicate and near-duplicate images can account for between 25 and 40 per cent of total image storage volume in large enterprise environments. For a mid-sized Singapore statutory board running an asset management platform, that translates directly into wasted expenditure on commercial cloud tiers. Amazon Web Services S3 storage in the Asia-Pacific Singapore region is priced from approximately USD 0.025 per gigabyte per month for standard access — small on a per-unit basis, but significant at scale when tens of terabytes of redundant files accumulate over years without automated deduplication.

The Housing and Development Board, which maintains photographic records for more than one million residential units across towns including Tampines, Bukit Merah, and Woodlands, is among the organisations that have been expanding digital inspection workflows since 2022. When field officers upload flat condition photographs from separate visits without a systematic hash-check at ingestion, identical or near-identical images stack up across job records. The same dynamic plays out at the Urban Redevelopment Authority, whose GeoSpace platform ingests street-level and aerial imagery on a rolling basis.

Singapore's National Library Board addressed a version of this problem when it digitised its heritage photograph collection at the National Archives on Canning Rise. Archivists found during a 2023 cataloguing exercise that a portion of scanned prints had been digitised twice — once from original negatives and once from reference prints — resulting in near-duplicate file pairs that required manual reconciliation before the records could be published on the BookSG and Roots.sg portals. The exercise underscored that deduplication is not purely a cost problem; it affects the integrity of public records.

Tools, Timelines, and What Comes Next

Perceptual hashing — a technique that generates a compact fingerprint from image content rather than its file metadata — has become the standard first-pass tool for large-scale duplicate detection. Libraries such as pHash and ImageHash can process thousands of images per second on modest hardware, and several Singapore-based managed service providers operating out of the one-north business park in Buona Vista have begun packaging deduplication pipelines as standalone managed services aimed at government accounts.

The practical roadmap for most organisations involves three stages: a baseline audit to establish the actual duplication rate, automated flagging at ingestion to prevent new duplicates entering the system, and a retrospective purge of confirmed redundant files after human or algorithmic review. Each stage carries its own cost and timeline. A baseline audit of a 50-terabyte image store typically takes between four and eight weeks, depending on whether the files are stored on-premises, on a hyperscaler platform, or spread across a hybrid environment — a common configuration for Singapore agencies that split workloads between government-managed GovTech infrastructure and commercial clouds.

Organisations that have not yet begun an audit should treat the IMDA's third-quarter data quality review window as a hard deadline for at least completing the baselining phase. Starting now, before storage bills compound further and before a compliance check surfaces the problem externally, remains the more cost-effective path.