1.2Data Validation and Processing Pipelines for FM Consumption

GenAI output quality depends on input data quality (garbage in, garbage out). Design pipelines that validate, format, and normalize data before it reaches the FM. Feature engineering itself is out of scope, but data preparation for FM consumption is tested.

1.2.1Validation and handling diverse data types

• AWS Glue Data Quality: define quality rules on datasets to catch missing/anomalous data before ingestion.
• SageMaker Data Wrangler / Processing: run preprocessing/transform jobs to shape data for FM input.
• Multimodal preprocessing: convert audio with Amazon Transcribe and documents/images with Amazon Textract to text before the FM.
• Input formatting: JSON structuring for the Bedrock API, dialog formatting, and entity extraction/normalization with Comprehend.

Before RAG or FM inference, reduce diverse source types to normalized text: Amazon Textract for scanned docs/PDFs (layout/tables/forms), Amazon Transcribe for calls/media (speaker diarization as needed), and Amazon Comprehend for structure extraction from free text (entities, key phrases, PII). Validate with AWS Glue Data Quality DQDL rules or custom Lambda checks, monitoring metrics in CloudWatch. Use SageMaker Processing/Data Wrangler for large transforms, Kinesis for streaming ingestion, and Glue jobs for batch. Finally, format into the JSON payload the Bedrock API expects (system prompt, messages array, parameters). Pre-processing low-quality input by rephrasing/summarizing with Bedrock itself is also effective.

1.2.2Section summary

• Quality gate = Glue Data Quality / docs = Textract / audio = Transcribe / extraction = Comprehend
• Pass FM input as structured JSON (optionally pre-clean low-quality input with Bedrock)