Mainframe Data Quality at Scale: Push Down What You Can, Stream Only What You Must

PDI pushdown-versus-stream decision matrix

PDI operating recommendation: choose the narrowest execution path that preserves the agreed rule semantics and fits the source’s resource budget. The matrix is a design aid, not a promise that every database, connector or PDI rule supports every SQL expression.

External platform fact: IBM’s Db2 for z/OS documentation lists COUNT, MIN, MAX, AVG and SUM aggregate functions, with function-specific type and null rules. That is evidence of supported Db2 SQL capabilities, not a claim of universal profiling performance, zero host cost or equivalent support in IMS, Adabas or other mainframe stores.

PDI operating safeguards before an approved run

• Workload windows: agree the source scope, read-only access, snapshot consistency and operating window with the platform owner; stop or defer work when the busy-host policy requires it.
• Resource limits: set supported time, row, transfer and concurrency budgets with the operator; define cancellation and partial-result handling rather than assuming a query is harmless.
• EXPLAIN and benchmark checks: have the Db2 owner inspect the access plan, then compare supported alternatives on an approved representative workload. An EXPLAIN plan is not a guarantee of elapsed time or production safety.
• Evidence retention: retain the SQL/rule version, approved scope, plan and benchmark references, execution window, exceptions and scorecard. Use approved access/retention controls; avoid retaining unnecessary raw personal rows.

Classify the workload before choosing the path

Start by translating each rule into the minimum information it needs. Counts, null totals, distinct counts, and length bounds may be expressible as aggregates in a supported SQL interface. Format validation, placeholder detection, or custom pattern checks may need selected column values evaluated by a rule engine. Some rules combine both approaches.

A workload inventory should state the table or segment, columns involved, predicate, aggregation or rule logic, expected output, and whether the logic can execute on the host. This prevents “pushdown” from becoming a blanket assumption and helps reviewers understand why particular data moved.

Use a rule-to-execution matrix

For each rule, record the datastore’s supported aggregate, required row-level inputs, expected result shape, access method, and execution window. If the platform cannot support the expression safely, mark that limit explicitly before selecting a streamed evaluation path.

Push down supported aggregates

Where the datastore and access layer support it, calculate appropriate aggregates on the host and return compact results. The mainframe can count rows, nulls, distinct values, or bounds without sending the complete table to the profiling client. This reduces transfer volume and keeps a large share of simple profiling work close to the data.

Pushdown must be validated against actual SQL support, data types, collation, null semantics, authorization, and query plans. A query that works in a development database may behave differently through a vendor gateway or a particular datastore interface. Capture the query or operation, parameters, execution identity, timestamp, and returned summary in the audit record.

Stream only the rules that need row-level values

When a rule needs values the host cannot evaluate in the required form, stream only the necessary columns and use bounded fetches. Do not fetch unrelated fields simply because they are available. Set explicit row and time budgets, use controlled scheduling, and agree on read semantics with the platform team. Read-only or uncommitted-read options are platform- and workload-specific decisions, not universal safety guarantees.

The profiler can then evaluate patterns or business rules off-host while retaining a record of rule version, rows considered, exclusions, and outcome counts. If sampling is used, label the result as sampled and record the sampling method. A sample-based estimate must not be presented as a full-population result.

Respect datastore-specific capability

The pushdown boundary varies across the mainframe estate. PDI’s published coverage describes full aggregate pushdown for Db2 z/OS, Db2 for i, and Datacom; aggregate support for IDMS and Adabas; and streamed evaluation for IMS. Validate those characteristics against the specific versions, connectivity, access path, and configuration in your environment before setting an operational expectation.

IMS is the clearest example of why a single architecture diagram can mislead: the published approach is streamed rather than aggregate pushdown. Other stores may expose different SQL or gateway behavior. An architecture review should therefore list each datastore, supported operations, data movement, dependencies, and fallback behavior separately.

Protect operational service while profiling

Work with system owners to schedule runs, establish workload limits, and monitor impact. Use least-privilege credentials and avoid unbounded scans during sensitive business windows. Agree how a run will stop if resource usage or latency exceeds an accepted threshold, and how operators will know whether it ended normally or was interrupted.

Do not assume a particular isolation mode is lock-free for every configuration. Concurrency effects depend on the datastore, access method, options, and workload. Test with the relevant platform team, document the chosen semantics and observed constraints, and treat “no locks” or “no impact” as a claim that requires precise evidence, not a default promise.

Make scorecards auditable

A scorecard is useful when each measure has a defined population, rule, and calculation. Identify the table and column, metric, rule registry or configuration identifier, execution path, source query or stream method, result, and limitations. Preserve enough detail to explain how a value was derived without unnecessarily retaining row-level sensitive data.

Use consistent dimensions such as completeness, validity, uniqueness, and consistency when they fit the rule. Keep raw counts alongside any normalized score so a reviewer can see the denominator. Record skipped columns, unsupported rules, timeouts, and access failures as explicit states rather than silently omitting them.

Pilot with evidence, not a throughput promise

Select a representative datastore and table, agree on a small set of pushdown and streamed rules, and capture the starting operational baseline. Compare returned aggregates with a trusted control where feasible; inspect streamed-rule samples with a data steward; and have the host team review workload effects. Record hardware, connectivity, data shape, configuration, and run conditions if performance is measured.

PDI’s mainframe data-quality page, whitepaper, and security resource provide product-specific detail for technical evaluation. Treat any published design or measurement as tied to its stated conditions, not a guarantee for every estate. A defensible rollout is the one that states what was evaluated, what ran where, what moved, what could not be measured, and who accepted the operational posture.