Data quality and management are key concerns for AI projects, survey finds

A recent survey conducted by Hitachi Vantara has highlighted significant issues concerning data quality and management in the context of artificial intelligence (AI) projects. The study, dubbed the State of Data Infrastructure Survey, gathered opinions from 1,200 IT decision-makers across 15 countries, revealing that high-quality data remains a fundamental requirement for successful AI implementation.

The survey revealed that 37% of IT leaders identified data as their foremost concern, a sentiment echoed by 41% of respondents in the United States, who pointed out that “using high-quality data” is the predominant factor contributing to the success of AI projects, both domestically and worldwide. As AI technology continues to proliferate, the demand for robust data infrastructure appears to be intensifying.

Moreover, Hitachi Vantara forecasts that the storage needs for data will surge by 122% by the year 2026. This projection raises alarming questions about the efficacy of current data management practices, particularly in light of the challenges already being faced by organisations. The findings indicate that while 38% of respondents have access to data most of the time, only 33% believe that the majority of their AI-generated outputs are accurate. Notably, 80% of the data surveyed is unstructured, complicating efforts to maintain quality as data volumes continue to escalate.

The report further unveiled troubling trends in how organisations are managing their data. Approximately 47% of surveyed companies do not implement tagging strategies for data visualisation, 37% are actively working on improving the quality of their training data, and a mere 26% bother to review their datasets for quality.

Amid these data management concerns, security has emerged as a top priority, with 54% of respondents citing it as their primary area of concern regarding their data infrastructure. A striking 74% acknowledged that significant data loss would be catastrophic for their operations, while 73% expressed apprehensions about potential hackers accessing AI-enhanced tools.

In terms of strategic development, sustainability and return on investment (ROI) seem to take a back seat. The study found that only 32% of IT decision-makers considered sustainability a top priority, while merely 30% were focusing on the ROI associated with their AI investments. Furthermore, a sizable 61% of larger corporations are concentrating on developing large language models (LLMs) instead of opting for smaller, more specialised models that could potentially require up to 100 times less power.

Simon Ninan, senior vice president of business strategy at Hitachi Vantara, commented on the situation, stating, “The adoption of AI depends very heavily on trust of users in the system and in the output. If your early experiences are tainted, it taints your future capabilities.” He added that many organisations are hastily pursuing AI initiatives without a clear strategy or specific outcomes, which could undermine the long-term success of such projects. Ninan emphasised the necessity of entering AI developments with well-defined use cases and ROI targets, alongside investing in modern infrastructure capable of managing large datasets efficiently while prioritising data resiliency and energy efficiency. He also cautioned that infrastructure developed without a focus on sustainability may require significant revisions to comply with future regulatory standards.

As AI technology continues to evolve rapidly, the insights from Hitachi Vantara’s survey underline the critical importance of addressing these foundational data challenges to harness the full potential of AI across sectors.

Source: Noah Wire Services