4Misc_Start@4Platform@@9VersionCheck xHH@Rg(HHdh xHH@Rg(HHdh x HHL Xg(HHdh ^Graph*WDashSettings#  ! 7<\6Normal@ Monaco<HHHH$$4 4 4 4 4 4 homefHdhd:Work:XOPDev:IgorXOPs7HR:HDF5:Examples:hdBDExamples cuHDF5(/:Work:XOPDev:IgorXOPs7HR:HDF5:Examples/Exampleshd%Work/XOPDev/IgorXOPs7HR/HDF5/Examples/+RecentWindows3D Graphics.ihfAdvanced Topics.ihfAnalysis.ihfAnnotations.ihfCurve Fitting.ihfData Folders.ihfDemoLoader.ipfDialog Help.ihfErrors.ihfFreeMem.ipfGetting Started.ihfGraphBrowser.ipfGraphics.ihfHDF5 Browser.ipfHDF5 Dataset LoaderNotes 4Misc_End@tXOPState_Start@@HDF5-64VISA64WindowXOP1-64IgorNeuronBridge-64SQL64IgorGIS64XTest3-64XTestStructs64XTest7-64PeakFunctions2-644XOPState_End@\!J  Packages!2   HDF5Browser!]  root_0!2ССmat0?????@@@@@0A@APA`ApAAAAAAABBB B$B(B,B0B4BTССwave0?????@@@@@  root_1!E2ССmat0?????@@@@ A0A@APA`AAAAAAAABBB B$B(B,B0BA TƷССwave0?????@@@@  root_2!1 ССmat0????@@@@A`ApAAAAAAAAAB BBBB0B4B8BCreate Demo HDF5 Files  This creates a folder named "Demo Threadsafe HDF5 Files" in your Igor Pro User Files folder and creates five HDF5 files in that folder. To see the folder and files, choose Help->Show Igor Pro User Files and inspect the "Demo Threadsafe HDF5 Files" folder.  %Each file consists of the following: / (root)  mat0 5x5 matrix  wave0 5x1 array  3Loading a Dataset from Each of a Set of HDF5 Files  9• Choose Macros->Demo Thread Safe HDF5 Dataset Loading  This loads a dataset named wave0 from each of the five HDF5 demo files into Igor's root data folder. The datasets are loaded into waves named wave0_0, wave0_1, wave0_2, and so on. If those waves already exist, they are overwritten.  The datasets are loaded by the thread worker function HDF5DatasetWorker in the "HDF5 Dataset Loader" procedure file. The DemoThreadSafeHDF5DatasetLoading procedure in that file orchestrates the threads that do the loading.  1Loading a Group from Each of a Set of HDF5 Files  9• Choose Macros->Demo Thread Safe HDF5 Dataset Loading  %This loads the root of each of the five HDF5 demo files into a separate data folder in Igor's root data folder. The root group from each of the five HDF5 demo files is loaded into data folders named root_0, root_1, root_2, and so on. If those data folders already exist, they are overwritten.  The groups are loaded by the thread worker function HDF5GroupWorker in the "HDF5 Group Loader" procedure file. The DemoThreadSafeHDF5GroupLoading procedure in that file orchestrates the threads that do the loading.  wThe rest of this file is intended for Igor programmers who want to use the techniques demonstrated in this experiment.  Igor Multitasking  $The code in the "HDF5 Dataset Loader" and "HDF5 Group Loader" procedure files is fairly complex. Before you examine it, you should familiarize yourself with Igor preemptive threading. See ThreadSafe Functions and Multitasking.  HThe technique used in this experiment passes data from the main thread and the preemptive thread and back using data folders as explained under Thread Data Environment and further explained under Input/Output Queues.  HDF5 Dataset Loader Details  _Open the "HDF5 Dataset Loader" procedure file as it is referenced in the following discussion.  The main function is named DemoThreadSafeHDF5DatasetLoading. It creates threads and sets the HDF5DatasetWorker function running on those threads. Although only one preemptive thread can access the HDF5 library at a time, the demo creates two threads as proof of concept.  AOne preemptive thread waits while another uses the HDF5 library. When the first thread finishes loading its file, the second thread proceeds to load its file. This continues until all files have been loaded at which time DemoThreadSafeHDF5DatasetLoading releases the thread group which terminates the preemptive threads.  After creating the threads, DemoThreadSafeHDF5DatasetLoading creates a data folder, called threadDF in this discussion and in the code, for each file to be loaded. It stores in threadDF parameters needed by the thread to identify the file to be loaded, the dataset to be loaded from that file, and the name to use for the output wave created by loading the dataset. It then calls ThreadGroupPutDF to move threadDF from the data hierarchy of the main thread to the thread group input queue from which a thread will retrieve it.  HDF5DatasetWorker threads call ThreadGroupGetDF to get a threadDF data folder from the thread group input queue. When a threadDF data folder is available, ThreadGroupGetDF returns it to HDF5DatasetWorker. HDF5DatasetWorker uses the parameters in threadDF to load a dataset from an HDF5 file and stores an output wave in threadDF. HDF5DatasetWorker then calls ThreadGroupPutDF to move threadDF to the thread group output queue for use by DemoThreadSafeHDF5DatasetLoading.  ^DemoThreadSafeHDF5DatasetLoading calls ThreadGroupGetDF to get a threadDF data folder from the thread group output queue. Such data folders become available when a given HDF5DatasetWorker thread finishes loading a file. DemoThreadSafeHDF5DatasetLoading moves the output wave from threadDF to the output data folder which is specified as a parameter.  "When DemoThreadSafeHDF5DatasetLoading has processed all a threadDF data folder for each HDF5 file to be loaded, all files have been loaded. DemoThreadSafeHDF5DatasetLoading calls ThreadGroupRelease which terminates the worker threads and releases the thread group. The process is finished.  For a dataset named wave0, the output waves are named wave0_0, wave0_1, and so on, so that a wave loaded from one HDF5 file does not conflict with a wave loaded from another HDF5 file.   HDF5 Group Loader Details  ]Open the "HDF5 Group Loader" procedure file as it is referenced in the following discussion.  The main function is named DemoThreadSafeHDF5GroupLoading. It creates threads and sets the HDF5GroupWorker function running on those threads. Although only one preemptive thread can access the HDF5 library at a time, the demo creates two threads as proof of concept.  ?One preemptive thread waits while another uses the HDF5 library. When the first thread finishes loading its file, the second thread proceeds to load its file. This continues until all files have been loaded at which time DemoThreadSafeHDF5GroupLoading releases the thread group which terminates the preemptive threads.  After creating the threads, DemoThreadSafeHDF5GroupLoading creates a data folder, called threadDF in this discussion and in the code, for each group to be loaded. It stores in threadDF parameters needed by the thread to identify the file to be loaded, the group to be loaded from that file, and the name to use for the output data folder created by loading the group. It then calls ThreadGroupPutDF to move threadDF from the data hierarchy of the main thread to the thread group input queue from which a thread will retrieve it.  HDF5GroupWorker threads call ThreadGroupGetDF to get a threadDF data folder from the thread group input queue. When a threadDF data folder is available, ThreadGroupGetDF returns it to HDF5GroupWorker. HDF5GroupWorker uses the parameters in threadDF to load a group from an HDF5 file and stores an output data folder in threadDF. HDF5GroupWorker then calls ThreadGroupPutDF to move threadDF to the thread group output queue for use by DemoThreadSafeHDF5GroupLoading.  gDemoThreadSafeHDF5GroupLoading calls ThreadGroupGetDF to get a threadDF data folder from the thread group output queue. Such data folders become available when a given HDF5GroupWorker thread finishes loading a file. DemoThreadSafeHDF5GroupLoading moves the output data folder from threadDF to the overall output data folder which is specified as a parameter.  When DemoThreadSafeHDF5GroupLoading has processed a threadDF data folder for each HDF5 file to be loaded, all files have been loaded. DemoThreadSafeHDF5GroupLoading calls ThreadGroupRelease which terminates the worker threads and releases the thread group. The process is finished.  For a group named Group0, the output data folders are named Group0_0, Group0_1, and so on, so that a data folder loaded from one HDF5 file does not conflict with a data folder loaded from another HDF5 file. Zx  " p x  . 6 HP|bj"08`h (0x| "* !!$$%%l't'((r)Stepl= X"S<,}M$HelveticaGenevaCourierMonacoTimes New RomanPalatinoTimesLucida GrandeSymbolOsakaITC Avant Garde GothicArialCourier NewAmerican Typewriter LightPlantagenet CherokeeOptimaMS PGothicLucida SansKhmer MNAmerican TypewriterArial Unicode MSZapf DingbatsLucida Sans Unicoded(d TXET0RGI:(HDF5 Group LoaderHDF5 Group Loader#pragma TextEncoding = "UTF-8" #pragma rtGlobals=3 // Use modern global access method and strict wave access. #pragma IgorVersion = 8.00 // These procedures requires Igor Pro 8.00 or later // HDF5GroupWorker(threadIndex, printProgress) // This is the thread worker function called by DemoThreadSafeHDF5GroupLoading to load an HDF5 group. // HDF5GroupWorker calls ThreadGroupGetDFR to get a reference to a data folder posted to // the thread group input queue by DemoThreadSafeHDF5GroupLoading. It then loads an HDF5 group // into that same data folder. Finally, it posts the same data folder to the thread group output // queue for use by DemoThreadSafeHDF5GroupLoading. ThreadSafe Function HDF5GroupWorker(int threadIndex, int printProgress) if (printProgress) Printf "HDF5GroupWorker starting, threadIndex=%d\r", threadIndex endif do do // Get free data folder from thread group input queue DFREF threadDF = ThreadGroupGetDFR(0,1000) if (DataFolderRefStatus(threadDF) != 0) // We got the free data folder and threadDF is a reference to it break else if (printProgress) if (GetRTError(2)) Print "HDF5GroupWorker closing down due to thread group release" else Print "HDF5GroupWorker still waiting for thread group input queue" endif endif endif while(1) // Create references to parameters passed to HDF5GroupWorker by DemoThreadSafeHDF5GroupLoading SVAR pathName = threadDF:gPathName SVAR fileName = threadDF:gFileName SVAR fullPathToFile = threadDF:gFullPathToFile SVAR groupNameOrPath = threadDF:gGroupNameOrPath // Name of or full HDF5 path to group to load - can be "/" to load entire file SVAR outputDataFolderName = threadDF:gOutputDataFolderName // gThreadResult is an output returned to DemoThreadSafeHDF5GroupLoading Variable/G threadDF:gThreadResult = 0 NVAR gThreadResult = threadDF:gThreadResult // Load the group from the HDF5 file Variable fileID = 0 do HDF5OpenFile /P=$pathName /Z fileID as fileName gThreadResult = V_Flag if (gThreadResult != 0) Printf "HDF5GroupWorker: HDF5OpenFile failed to load file \"%s\", result=%d\r", fullPathToFile, gThreadResult break // Error endif HDF5LoadGroup /O /T=$outputDataFolderName /Q /Z threadDF, fileID, groupNameOrPath gThreadResult = V_Flag if (gThreadResult != 0) Printf "HDF5GroupWorker: HDF5LoadGroup failed to load group \"%s\" from file \"%s\", result=%d\r", groupNameOrPath, fullPathToFile, gThreadResult break // Error endif while(0) if (fileID != 0) HDF5CloseFile fileID endif if (gThreadResult == 0) // Load succeeded? // Move the output data folder to threadDF which will soon be output data folder DFREF outputDataFolder = $outputDataFolderName MoveDataFolder outputDataFolder, threadDF if (printProgress) Printf "HDF5GroupWorker loaded group \"%s\"\r", fullPathToFile endif else Printf "HDF5GroupWorker failed to load file \"%s\", result=%d\r", fullPathToFile, gThreadResult endif outputDataFolder = $"" // Without this, ThreadGroupPutDF would fail // Convert threadDF from free to normal. This is needed to prevent the // data folder from being killed when threadDF is set to NULL ($"") // without which ThreadGroupPutDF would fail. MoveDataFolder threadDF, : SetDataFolder threadDF threadDF = $"" // Without this, ThreadGroupPutDF would fail ThreadGroupPutDF 0, : // Move threadDF to thread group output queue while(1) return 0 End // DemoThreadSafeHDF5GroupLoading(pathName, groupNameOrPath, outputDF, verbosity) // // Demonstrates loading a group from each file of a set of HDF5 files in a preemptive thread. // // pathName is the name of an Igor symbolic path pointing to a folder on disk // that contains one or more HDF5 files with ".h5" extensions. // // groupNameOrPath is the name of or full HDF5 path to the group to load from each HDF5 file. // Use "/" for the HDF5 file root, not ".". // // outputDF is the data folder in which output data folders are to be stored. // The output data folder must already exist. Pass "root:" to use the root data folder. // // verbosity is a bitwise parameter defined as follows: // Bit 0: Emit progress reports from main function (DemoThreadSafeHDF5GroupLoading) // Bit 1: Emit progress reports from worker function (HDF5GroupWorker) // Error messages are emitted regardless of the value of verbosity. // // The data folders loaded from the HDF5 files are returned in the current data folder. // The output data folder names are given suffixes so that the don't conflict with each other. // For example, if we are loading a group named "Group0", the output waves are // Group0_0, Group0_1, ... Function DemoThreadSafeHDF5GroupLoading(pathName, groupNameOrPath, outputDF, verbosity) String pathName // Name of Igor symbolic path containing HDF5 files String groupNameOrPath // Name of or full HDF5 path to group to load from each file DFREF outputDF // The output data folder in the main thread's data hierarchy Variable verbosity // See comments above PathInfo DemoFiles if (V_Flag == 0) Abort "Choose Macros->Create Demo HDF5 Files first." endif if (DataFolderRefStatus(outputDF) == 0) Abort "DemoThreadSafeHDF5GroupLoading: Output data folder does not exist" endif int printProgressMain = (verbosity & 1) != 0 int printProgressWorker = (verbosity & 2) != 0 PathInfo $pathName if (V_flag == 0) Abort "Invalid symbolic path name" endif String fullPathToFolder = S_path // Get list of files to load String fileList = IndexedFile($pathName, -1, ".h5") // List of all .h5 files in directory Variable numFilesTotal = ItemsInList(fileList) Variable numFilesSuccessfullyLoaded = 0 Variable numFilesThatFailed = 0 Variable numFilesProcessed = 0 // numFilesSuccessfullyLoaded + numFilesThatFailed // As of this writing, the HDF5 library can load only one file at a time. // If it is busy loading one file, it blocks loading of additional files until // the first load finishes. Consequently there is currently no need to use // more than one thread. However, as proof of concept, we use two threads. Variable numThreads = 2 // Create threads Variable threadGroupID = ThreadGroupCreate(numThreads) Variable i for(i=0; i 0) Printf "DemoThreadSafeHDF5GroupLoading failed to load %d files because of errors\r", numFilesThatFailed endif End "(d TXET0RGI'HDF5 Dataset LoaderHDF5 Dataset Loader#pragma TextEncoding = "UTF-8" #pragma rtGlobals=3 // Use modern global access method and strict wave access. #pragma IgorVersion = 8.00 // These procedures requires Igor Pro 8.00 or later // HDF5DatasetWorker(threadIndex, printProgress) // This is the thread worker function called by DemoThreadSafeHDF5DatasetLoading to load an HDF5 file. // HDF5DatasetWorker calls ThreadGroupGetDFR to get a reference to a data folder posted to the // thread group input queue by DemoThreadSafeHDF5DatasetLoading. It then loads an HDF5 file // into that same data folder. Finally, it posts the same data folder to the thread group output // queue for use by DemoThreadSafeHDF5DatasetLoading. ThreadSafe Function HDF5DatasetWorker(int threadIndex, int printProgress) if (printProgress) Printf "HDF5DatasetWorker starting, threadIndex=%d\r", threadIndex endif do do // Get free data folder from thread group input queue DFREF threadDF = ThreadGroupGetDFR(0,1000) if (DataFolderRefStatus(threadDF) != 0) // We got the free data folder and threadDF is a reference to it break else if (printProgress) if (GetRTError(2)) Print "HDF5DatasetWorker closing down due to thread group release" else Print "HDF5DatasetWorker still waiting for thread group input queue" endif endif endif while(1) // Create references to parameters passed to HDF5DatasetWorker by DemoThreadSafeHDF5DatasetLoading SVAR pathName = threadDF:gPathName SVAR fileName = threadDF:gFileName SVAR fullPathToFile = threadDF:gFullPathToFile SVAR datasetNameOrPath = threadDF:gDatasetNameOrPath // Name or full path to dataset to load SVAR outputWaveName = threadDF:gOutputWaveName // gThreadResult is an output returned to DemoThreadSafeHDF5DatasetLoading Variable/G threadDF:gThreadResult = 0 NVAR gThreadResult = threadDF:gThreadResult // Load the dataset from the HDF5 file Variable fileID = 0 do HDF5OpenFile /P=$pathName /Z fileID as fileName gThreadResult = V_Flag if (gThreadResult != 0) Printf "HDF5DatasetWorker: HDF5OpenFile failed to load file \"%s\", result=%d\r", fullPathToFile, gThreadResult break // Error endif HDF5LoadData /O /N=$outputWaveName /Q /Z fileID, datasetNameOrPath gThreadResult = V_Flag if (gThreadResult != 0) Printf "HDF5DatasetWorker: HDF5LoadData failed to load dataset \"%s\" from file \"%s\", result=%d\r", datasetNameOrPath, fullPathToFile, gThreadResult break // Error endif while(0) if (fileID != 0) HDF5CloseFile fileID endif if (gThreadResult == 0) // Load succeeded? // Move the output wave to threadDF which will soon be output data folder Wave outputWave = $outputWaveName MoveWave outputWave, threadDF if (printProgress) Printf "HDF5DatasetWorker loaded file \"%s\"\r", fullPathToFile endif else Printf "HDF5DatasetWorker failed to load file \"%s\", result=%d\r", fullPathToFile, gThreadResult endif WaveClear outputWave // Without this, ThreadGroupPutDF would fail // Convert threadDF from free to normal. This is needed to prevent the // data folder from being killed when threadDF is set to NULL ($"") // without which ThreadGroupPutDF would fail. MoveDataFolder threadDF, : SetDataFolder threadDF threadDF = $"" // Without this, ThreadGroupPutDF would fail ThreadGroupPutDF 0, : // Move threadDF to thread group output queue while(1) return 0 End // DemoThreadSafeHDF5DatasetLoading(pathName, datasetNameOrPath, outputDF, verbosity) // // Demonstrates loading a dataset from each file of a set of HDF5 files in a preemptive thread. // // pathName is the name of an Igor symbolic path pointing to a folder on disk // that contains one or more HDF5 files with ".h5" extensions. // // datasetNameOrPath is the name of or full HDF5 path to the dataset to load from each HDF5 file. // // outputDF is the data folder in which output waves are to be stored. // The output data folder must already exist. Pass "root:" to use the root data folder. // // verbosity is a bitwise parameter defined as follows: // Bit 0: Emit progress reports from main function (DemoThreadSafeHDF5DatasetLoading) // Bit 1: Emit progress reports from worker function (HDF5DatasetWorker) // Error messages are emitted regardless of the value of verbosity. // // The waves loaded from the HDF5 files are returned in the current data folder. // The output wave names are given suffixes so that the don't conflict with each other. // For example, if we are loading a dataset named "wave0", the output waves are // wave0_0, wave0_1, ... Function DemoThreadSafeHDF5DatasetLoading(pathName, datasetNameOrPath, outputDF, verbosity) String pathName // Name of Igor symbolic path containing HDF5 files String datasetNameOrPath // Name of or full HDF5 path to the dataset to load from each file DFREF outputDF Variable verbosity // See comments above PathInfo DemoFiles if (V_Flag == 0) Abort "Choose Macros->Create Demo HDF5 Files first." endif if (DataFolderRefStatus(outputDF) == 0) Abort "DemoThreadSafeHDF5DatasetLoading: Output data folder does not exist" endif int printProgressMain = (verbosity & 1) != 0 int printProgressWorker = (verbosity & 2) != 0 PathInfo $pathName if (V_flag == 0) Abort "Invalid symbolic path name" endif String fullPathToFolder = S_path // Get list of files to load String fileList = IndexedFile($pathName, -1, ".h5") // List of all .h5 files in directory Variable numFilesTotal = ItemsInList(fileList) Variable numFilesSuccessfullyLoaded = 0 Variable numFilesThatFailed = 0 Variable numFilesProcessed = 0 // numFilesSuccessfullyLoaded + numFilesThatFailed // As of this writing, the HDF5 library can load only one file at a time. // If it is busy loading one file, it blocks loading of additional files until // the first load finishes. Consequently there is currently no need to use // more than one thread. However, as proof of concept, we use two threads. Variable numThreads = 2 // Create threads Variable threadGroupID = ThreadGroupCreate(numThreads) Variable i for(i=0; i 0) Printf "DemoThreadSafeHDF5DatasetLoading failed to load %d files because of errors\r", numFilesThatFailed endif End // Platform=Macintosh, IGORVersion=8.000, architecture=Intel, systemTextEncoding="MacRoman", historyTextEncoding="MacRoman", procwinTextEncoding="UTF-8", recreationTextEncoding="UTF-8", build=30432 #pragma TextEncoding = "UTF-8" Silent 101 // use | as bitwise or -- not comment. DefaultFont "Helvetica" String/G root:gWMSetNextTextFilesTextEncoding = "UTF-8" // Text encoding for "HDF5 Dataset Loader". Used by Igor Pro 7. OpenProc/W=(45,58,1182,875)/J=30931/V=0 "HDF5 Dataset Loader" String/G root:gWMSetNextTextFilesTextEncoding = "UTF-8" // Text encoding for "HDF5 Group Loader". Used by Igor Pro 7. OpenProc/W=(46,49,1094,873)/J=20471/V=0 "HDF5 Group Loader" MoveWindow/P 60,55,1189,860 MoveWindow/C 4,640,1280,1024 String/G root:gWMSetNextTextFilesTextEncoding = "MacRoman" // Text encoding for Notes. Used by Igor Pro 7. OpenNotebook/N=Notes/W=(133,48,889,611)/J=8797 "Notes" KillStrings/Z root:gWMSetNextTextFilesTextEncoding T#pragma TextEncoding = "UTF-8" #pragma rtGlobals=3 // Use modern global access method and strict wave access. Menu "Macros" "Create Demo HDF5 Files/0", CreateDemoHDF5Files(5) "Demo Thread Safe HDF5 Dataset Loading/1", DemoThreadSafeHDF5DatasetLoading("DemoFiles", "wave0", root:, 1) // Loads dataset "wave0" from root of each HDF5 file "Demo Thread Safe HDF5 Group Loading/2", DemoThreadSafeHDF5GroupLoading("DemoFiles", "/", root:, 1) // Loads root of each HDF5 file End Function CreateDemoHDF5Files(int numFiles) PathInfo IgorUserFiles String fullPathToFolder = S_path + "Demo Threadsafe HDF5 Files" NewPath/O/C/Q DemoFiles, fullPathToFolder DFREF dfr = NewFreeDataFolder() int i for(i=0; i