I/O Servers
Friends and colleagues approached me about a month or so ago with a request to integrate a service program of their design into a basic database inquiry and maintenance application of mine. They dubbed their service program an "I/O server".
These folks are engaged in an education-marketing campaign, fervently urging shops to replace RPG I/O op codes that have traditionally been embedded in business applications, with calls to I/O servers instead, under the auspices of "modernizing" database I/O.
The idea is to create an I/O server for every physical and logical file in your database, and make calls to I/O server procedures instead of embedding RPG op codes within applications. A tool may be licensed that automates the generation of an I/O servers from file meta-data, based on RPG templates.
The procedural interface for an I/O server consists of three (3) parameters. The first two parameters are required. The third parameter is optional based on whether the requested I/O calls for a record key.
- Action Code (e.g. I=Insert, U=Update, D=Delete, R=Read, S=Setll, G=Setgt, etc.).
- A pointer to a record buffer that is shared between the application and the I/O server.
- A data structure that contains the key(s) to a file.
Following is a sample source-code module for this type of I/O server:
h nomain aut(*use) option(*nodebugio) debug bnddir('AMWSRV')
fcusmstf uf a e k disk usropn commit rename(cusmstfr:rec)
d cusmstfp s *
d cusmstfk e ds extname(cusmstf:*key) qualified
d cusmstfr e ds extname(cusmstf) based(cusmstfp)
d cusmstio pr n
d action 5 const
d pointer * const
d keys const options(*nopass) like(cusmstfk)
* move *diag & re-send *escape messages (percolate)
d err03 pr
d mt 1 const options(*nopass)
* send *escape message
d err11 pr
d mi 7 const
d mf 10 const options(*omit:*nopass)
d md 128 const options(*omit:*nopass)
d mt 1 const options(*nopass)
d put pr n
d upd pr n
d dlt pr n
d sets pr n
d setg pr n
d get pr n
d getnxt pr n
d getprv pr n
d savePtr s * inz(%addr(saverec))
d saverec ds likerec(rec)
d saveKey ds likerec(rec:*key)
d saveKey# s 5i 0
d action ds 5
d a1 1
d a2 1
d keys s 5i 0
// ----------------------------------------------------------------
// perform I/O on the customer master file
// ----------------------------------------------------------------
p cusmstio b export
d cusmstio pi n
d A 5 const
d P * const
d K const options(*nopass) like(cusmstfk)
/free
monitor;
if not %open(cusmstf);
open cusmstf;
endif;
action = A;
cusmstfp = P;
select;
when %parms = 2 and a1 = 'D';
eval-corr cusmstfk = cusmstfr;
when %parms = 3;
cusmstfk = K;
endsl;
select;
when a1 = 'I'; // Insert
return put();
when a1 = 'U'; // update
return upd();
when a1 = 'D'; // Delete
return dlt();
when a1 = 'R'; // Read
return get();
when a1 = 'S'; // Set Limits (SETLL)
return sets();
when a1 = 'G'; // Set Limits (setgT)
return setg();
when a1 = 'X'; // Close
close cusmstf;
*inlr = *on;
return *on;
other;
err11('IOS9999':'AMWMSGF':'cusmstio '+action);
endsl;
on-error;
err03();
endmon;
/end-free
p cusmstio e
// ----------------------------------------------------------------
// write a record
// ----------------------------------------------------------------
p put b
d put pi n
/free
monitor;
setll *hival cusmstf;
readp(n) cusmstf saverec;
if %eof;
custno = 1;
else;
custno = saverec.custno + 1;
endif;
strdte = %date();
write rec;
return *on;
on-error;
err03();
endmon;
/end-free
p put e
// ----------------------------------------------------------------
// update a record
// ----------------------------------------------------------------
p upd b
d upd pi n
/free
monitor;
eval-corr cusmstfk = cusmstfr;
chain %kds(cusmstfk) cusmstf saverec;
select;
when %found;
update rec;
return *on;
when a2 = 'I';
return put();
other;
return *off;
endsl;
on-error;
err03();
endmon;
/end-free
p upd e
// ----------------------------------------------------------------
// delete a record
// ----------------------------------------------------------------
p dlt b
d dlt pi n
/free
monitor;
delete %kds(cusmstfk) cusmstf;
return *on;
on-error;
err03();
endmon;
/end-free
p dlt e
// ----------------------------------------------------------------
// set lower limit / check existence
// ----------------------------------------------------------------
p sets b
d sets pi n
/free
monitor;
setll %kds(cusmstfk) cusmstf;
return %equal;
on-error;
err03();
endmon;
/end-free
p sets e
// ----------------------------------------------------------------
// set grater than / check existence
// ----------------------------------------------------------------
p setg b
d setg pi n
/free
monitor;
setgt %kds(cusmstfk) cusmstf;
return %found;
on-error;
err03();
endmon;
/end-free
p setg e
// ----------------------------------------------------------------
// retrieve a record
// ----------------------------------------------------------------
p get b
d get pi n
/free
monitor;
select;
when a2 = ' '; // Read - All keys
chain(n) %kds(cusmstfk) cusmstf;
return %found;
when a2 = 'N'; // Read Next
return getnxt();
when a2 = 'P'; // Read Previous
return getprv();
when a2 = 'F'; // Read Next record
read(n) cusmstf;
when a2 = 'B'; // Read Previous record
readp(n) cusmstf;
endsl;
return %eof;
on-error;
err03();
endmon;
/end-free
p get e
// ----------------------------------------------------------------
// read next equal - don't lock
// ----------------------------------------------------------------
p getnxt b
d getnxt pi n
/free
monitor;
reade(n) %kds(cusmstfk) cusmstf;
return %eof;
on-error;
err03();
endmon;
/end-free
p getnxt e
// ----------------------------------------------------------------
// read previous equal - don't lock
// ----------------------------------------------------------------
p getprv b
d getprv pi n
/free
monitor;
readpe(n) %kds(cusmstfk) cusmstf;
return %eof;
on-error;
err03();
endmon;
/end-free
p getprv e
Is this type of I/O interface familiar to you?
My initial thought for this piece was to delineate the pros and cons of replacing RPG I/O op codes in applications with calls to I/O servers. However, as I studied the architecture implemented in this type of I/O server, I couldn't come up with any pros.
Virtually every point I could think of, favored the embedding of RPG I/O op codes in applications, as opposed to invoking procedures in I/O servers from applications.
After reviewing the I/O-server source code in detail, one can see that the execution path entails a number of steps before an actual I/O op code is invoked:
- Two (2) procedure calls (the entry-point procedure, plus one pertaining to requested actions).
- Two (2) monitor, on-error blocks (embedded in procedures).
- Two to three (2-3) select blocks (based on requested actions)
The I/O server interface is designed to percolate I/O related *diagnostic messages, and to re-send *escape messages up one level in the call-stack, as opposed to say something like transforming them into a more friendly format.
When opening a file in an I/O server, there is no option for selecting an open mode (e.g. input, update, output, overwrite, block buffering, etc.). Rather the file is opened in a mode that provides for all I/O operations and record-lock options.
The commit keyword on the "f" spec designates the use of commitment control (applications and I/O servers must run under commitment control).
What sort of impact might an elongated execution path, structure, and commitment control have on I/O performance? The following benchmarks show reading the CUSMSTF file front to back, first using the I/O server, then next by just embedding the "f" spec and read op codes in the program.
Read CUSMSTF file using I/O Server:
h bnddir('AMWSRV')
d cusmstfp s * inz(%addr(cusmstfr))
d cusmstfr e ds extname(cusmstf)
d ok s n inz(*off)
d cusmstio pr n
d Action 5 const
d Pointer * const
d Keys const options(*nopass) like(cusmstfk)
/free
dow ok = *off;
ok = cusmstio('RF' : cusmstfp);
enddo;
*inlr = *on;
return;
/end-free
Read CUSMSTF file using the read op code:
fcusmstf if e k disk
/free
read cusmstfr;
dow not %eof();
read cusmstfr;
enddo;
*inlr = *on;
return;
/end-free
Given 600,000 rows in the CUSMSTF file, the call to the CUSMSTIO() procedure in a loop consumes 5.6 times more CPU than invoking the READ op code within a loop and checking for %eof().
Questioning the poor performance of the CUSMSTIO() procedure, one concerned reader asked whether the I/O server was running in a *new activation group, or whether the I/O involved opening and closing the file repeatedly?
The answer is NO on both counts. CUSMSTIO() runs in the *caller activation group, and the CUSMSTF file is opened only once.
The lag in READ performance in this case is mostly due to the I/O server being unable to take advantage of block mode processing. Other benchmarks suggest that the I/O server code structure adds 35-40 percent of unnecessary overhead to I/O op codes.
Given the lag in performance when using an I/O server, one might ask whether the performance might be offset by an improvement in application architecture. The answer is NO! This type of I/O server is problematic throughout.
A number of common use-cases call for multiple instances of the same file to be opened within the same Job (occasionally multiple instances of the same file to be opened within the same program), and for each instance of the file to have its own open data path.
An I/O server provides a SHARED open data path for any and all programs that are running in the same activation group. That would require work-a rounds in a number of use cases.
Some database designs implement multi-member tables. Big data is a possible use case. However the I/O server interface has no provision for members (partitions in Db2 parlance).
I/O servers are not designed to open multiple concurrent instances of files from different libraries.
The I/O server referenced in this piece implements just a small number of record positioning, READ, LOCK, and KEY options. In order to implement a full compliment of such options, including partial key options would greatly increase the amount of code required, the number of internal procedures encapsulated, and the number of SELECT block options therein. Actually, using partial keys is not even an option with the given interface (although it may be required by some use cases).
Record positioning and reading may involve criteria such as:
- First.
- Last.
- Next.
- Prior.
- By relative record number.
- Equal to.
- Less than.
- Greater than.
- Less than or equal to.
- Greater than or equal to.
- Next equal to.
- Prior equal to.
- Next unequal to.
- Prior unequal to.
- Beginning of file.
- End of file.
- Etc.
A full complement of Action options would be quite extensive; should arguably be implemented as named constants and kept in a /copy member. I/O server actions may be invoked from many programs.
Encapsulating a full compliment of I/O procedures in this manner entails a lot of repeated structure, including repeated boiler-plate code, which is an aberration of the Don't-Repeat-Yourself principle of application design.
Any change to the layout of a record format in an I/O server would require recompiling the I/O server, plus recompiling every other module that referenced the data buffer. Otherwise it is probable that invalid data would be inserted into records, and you wouldn't even get the benefit of a file-level-check error.
Error monitoring and message percolation up the call stack in an I/O server is redundant, in that the calling modules must also monitor for the same.
Setting up commitment control is typically very disruptive to legacy applications. It's arguably not needed for single-record updates as implemented in I/O servers, but rather designed to ensure the integrity of multi-update transactions. Existing applications would be forced to implement commitment boundaries, commits, and rollbacks (not trivial).
Application architects often promote the idea of encapsulating and "externalizing" data validation, referential integrity constraints, business rules, mapping between business objects and database objects, and other functionality that may require database I/O. That may be the well-meaning motivation for promoting this type of I/O server.
However, that is NOT what these I/O servers are designed to do. The interface in this I/O-server implementation is constrained to the three (3) parameters that I referenced previously. These I/O servers merely provide a procedure interface that wraps RPG I/O op codes, and monitors for errors that may be returned by Db2 for i, or errors that may be returned by trigger programs.
Although some business logic could be included in this type of I/O server, it would most likely be better suited if placed outside (perhaps in a trigger program, or in a calling program). An example of such might be the logic for automatically generating a customer number in the CUSMSTF file.
This type of I/O server fits the definition of an anti-pattern. It may look attractive on the surface. It may work in a narrow context. But it turns out badly when applied in a broader context. One reader suggested that this type of interface is regressive.
In a future article, I may address how to gain the benefits of externalizing I/O related logic without the overhead, constraints, redundancy, and other problems associated with this type of I/O server.