Server Disk Logging in Data Acquisition
Technical Note TN-232 Version 1.1
DAPcell Server version 4.00 - included with the DAPtools Standard and DAPtools Professional software bundles - introduced two additional DAPIO32 services for high-speed data transfers: DapPipeDiskLog and DapPipeDiskFeed. DapPipeDiskLog streams data directly from a DAP or iDSC board to any file you choose for postprocessing in any application. In the opposite direction, DapPipeDiskFeed pulls data directly from a disk file onto a DAP or iDSC for onboard processing.
Microstar Laboratories built into these new PC services the same technology used in DAPL: optimized multithreaded tasks for handling complex buffers and other data structures, all simply defined by nonthreaded user code. The high efficiency of the task - disk I/O - remains unchanged whatever the calling environment: C, Visual Basic, or some other Windows application.
With the proper configuration of DAP and PC hardware, the new disk I/O services can sustain data transfers indefinitely at a rate of 8.2M samples per second¹ or more. Very high speed logging requires very high speed PC components: CPU, RAM, motherboard and hard drive. Extensive memory helps, as well.
DapPipeDiskLog and DapPipeDiskFeed operate as background tasks: one initial setup defines what each service performs in any application. The services can transfer data to and from any file on any drive on any PC on a network. Network capacity may limit data transfer rates to and from remote PCs.
Example - setting up DapPipeDiskLog
This C example (taken from the DAPIO32 reference manual) shows how easily you can start a data logging session by using DapPipeDiskLog. The setup for DapPipeDiskFeed looks much the same.
/* Set up the structure containing the disk log information */
TDapPipeDiskLog LogInfo;
DapStructPrepare(&LogInfo, sizeof(LogInfo));
LogInfo.bmFlags = dpdl_ServerSide | dpdl_FlushBefore;
LogInfo.pszFileName = "c:\\logfiles\\logfile.log";
LogInfo.dwOpenFlags = DAPIO_CREATE_ALWAYS;
/* Open a log handle */
hLog = DapHandleOpen( "\\\\PC92\\Dap3\\$BinOut", DAPOPEN_DISKIO);
/* Start a logging session */
if ( hLog && DapPipeDiskLog(hLog, &LogInfo, NULL) )
{
/* The thread is now logging data in the background. */
/* Logging status can be checked periodically */
/* if desired, or this calling task can go to sleep. */
}
else
{
/* handle error */
}
/* Terminate the logging session */
if (hLog) DapHandleClose(hLog);
Real-World Application - high-speed logging for postprocessing
Microstar Laboratories developed the DapPipeDiskLog and DapPipeDiskFeed services to meet the following design requirements:
- 64 channels per scalable unit
- Up to 153.6K samples per second per channel - the maximum for the iDSC
- Test configuration: two scalable units, each with eight iDSCs, each to use all of its eight input channels (2 x 8 x 8 = 128) for a total of two, fully loaded, scalable units
- Test sampling rate per scalable unit: 8 x 8 x 153.6K = 9.8304M samples per second
- Indefinitely sustained data transfers without iDSC overflow
- Remote control of services across a network
- Simultaneous back-up of all data to a separate disk drive (mirroring)
The first customer shipment closely matched the test configuration described above, and provided 128 channels of synchronized simultaneous data acquisition continuously logged to disk. When, after 30 minutes, the system ran out of disk space, it had logged 30 x 60 x 2 x 9.8304M = 35.389G samples. Preliminary tests of scalability indicate that as many as 16 units can work together to provide up to 1024 channels of synchronized simultaneous data acquisition continuously logged to disk.
Sample Application - arbitrary waveform generation
You can use the DapPipeDiskFeed service to send any data from a disk file to the DAP. You can download raw data or waveform parameters to the DAP for any purpose - for example to perform subsequent processing, or to control some electromechanical device. An option in DapPipeDiskFeed allows you to create an arbitrary periodic waveform by repeating the data continuously.
Example: a mathematically hard-to-describe waveform
Only one cycle of this wave, digitized and saved to a file, forms the data set that DapPipeDiskFeed repeatedly sends to a DAP. DapPipeDiskLog could have saved the data from a DAP in the first place to provide the source used here - after waveform averaging, maybe, or some other postprocessing.
Sample Application - delayed analysis of test results
Many customers use software such as MATLAB or EXCEL to analyze the data captured by a DAP. Sufficiently complex analysis may prevent these Windows-based products from keeping up with the input at run-time. This causes DAP input buffers to overflow, and that stops input sampling. Using DapPipeDiskLog solves this problem. MATLAB or other analysis software then can take as much time as needed to process the logged data.
To perform subsequent analysis in DAPL, customers can simply transfer logged data back to the DAP using DapPipeDiskFeed.
Sample Application - logging data from multiple channels
Multithreaded design makes DapPipeDiskLog a safe and simple choice for logging data from several channels with mixed rates. Previously available services, like DapBufferGetEx, could log mixed-rate data from multiple channels, but only with careful provision to avoid data overflow. DapPipeDiskLog can log mixed-rate data from multiple channels safely and easily. The following listing compares the two approaches.
Open handles and set up the Dapio structures
while (not done)
{
DapBufferGetEx ( channel 1) // cannot proceed until
// minimum data requirement
// is met
Write channel 1 buffer to disk
DapBufferGetEx ( channel 2) // data continues to
// accumulate in channel 2
Write channel 2 buffer to disk
.. and so on
}
Close all handles and stop.
Now consider the code used with DapPipeDiskLog - still waiting on channel 1
Using DapPipeDiskLog, the structure becomes:
Open handles and set up the Dapio structures
DapPipeDiskLog ( channel 1 ) // set up thread and
// return, no waiting
DapPipeDiskLog ( channel 2 ) // continue with
// execution immediately
... and so on
while (not done) { Sleep() } // or other processing
Close all handles and stop
¹ A single DAPcell server PC can support up to 14 DAP or iDSC boards as standard. In a benchmark server configured to match customer hardware for the first shipment of DAPcell Server version 4.00, Microstar Laboratories used 4 boards, each with an overall data acquisition rate of 1228.8k samples per second, and logged to disk at the full acquisition rate: 4.9152M samples per second. With DAPcell Network Server, a single DAPcell client PC, requesting the disk logging service from 11 servers configured like the benchmark server, then can log to disk at an overall rate of 54.0672M samples per second.
In a more recent benchmark, Microstar Laboratories used 3 DAP boards, each with an overall data acquisition rate of 1667k samples per second, or 5001k samples per second for all three. The server PC logged 5M samples per second with capacity to spare. On any DAP board, running the DAPL REPLICATE command on the onboard processor increases the apparent sample throughput. Doing this on just two of the boards � or on all three � established 8.2M samples per second as the maximum sustained disk-logging rate on the benchmark PC hardware. With DAPcell Network Server, a single DAPcell client PC, requesting the disk logging service from 11 servers configured like the benchmark server, then can log to disk at an overall rate of more than 90M samples per second.
View other Technical Notes.