Share story
Infrastructure

AWS adds native data validation to EMR Spark jobs at no additional cost

AWS says Amazon EMR now supports data quality checks directly within Apache Spark jobs at no additional cost. The feature lets data engineering teams validate records during processing and fail pipelines, filter invalid records or route them to quarantine based on the results. Powered by AWS Glue, it accepts rules written in Spark scripts or stored in the Glue Data Catalog. More than 30 rule types include comparisons with historical runs and machine learning detection of data drift without fixed thresholds. Results identify failures at dataset, rule and row level. The feature supports Apache Iceberg, Apache Hudi and Delta Lake across EC2, EKS and Serverless deployments. Availability requires AWS runtime for Apache Spark emr-spark-8.1.0 in commercial and US GovCloud regions where EMR operates.
Sources
Recent Announcements
In this story
Amazon
Published by Tech & Business, a media brand covering technology and business. This story was sourced from Recent Announcements and reviewed by the T&B editorial agent team.
Back to Newswire
Keep reading
Full wire
Products Security
Products Security

IonQ agrees to acquire Elisity for up to $1.3 billion in stock

Quantum computing firm IonQ has agreed to acquire Elisity for as much as $1.3 billion in an all-stock transaction, Bloomberg reported. The deal is intended to expand IonQ's cybersecurity platform to address dangers assisted by art...

Bloomberg Technology
AI Capital
AI Capital

Suki raises $90 million to expand healthcare AI workflows

Suki raised a $90 million Series E led by Healthier Capital to scale its healthcare AI scribe business and add workflows in which AI takes actions, CEO Punit Soni told Axios. The company provides AI tools that document care for he...

Techmeme
Security Policy
Security Policy

UK expands AI vulnerability searches into closed government systems

The UK government will expand its use of frontier AI models to find vulnerabilities, moving the next phase into closed government systems. One pilot generated 407 findings across nine government organisations during a month of wee...

The Next Web
AI Infrastructure
AI Infrastructure

AWS launches Ray LLM containers for inference across GPU nodes

AWS has launched Ray LLM Deep Learning Containers for serving language models across multiple GPU nodes, extending its preassembled inference stack beyond single-node deployments. The containers bundle the inference engine, servin...

Artificial Intelligence
Infrastructure AI
Infrastructure AI

Microsoft details 96-chip Maia 300 racks and expanded inference network

Microsoft detailed a Maia 300 design that doubles accelerator density to 96 chips in a 167-kW rack and connects up to 12,288 accelerators through one network. The company says the full rack delivers up to 7 times Maia 200's token ...

azure.microsoft.com