Imaris Open - Discussion Forum

This is a community discussion space where Imaris is the key topic. This forum will offer guidance to new users, technical as well as application support. Users from sites with a valid Imaris Infinity subscription will receive priority – however, the fastest way to get to the Bitplane Technical and Application Support team will continue to be via Asia, EU or US support mailboxes. Every registered user can add items to the forum. Non-registered users can read the forum but cannot add/comment, etc. All ideas, questions, suggestions and comments are welcome.

Advanced Search
Thread Author
Max. Age of Thread
Number of Posts
Thread Rating
HomeHomeXTensionsXTensionsMatlabMatlabBitplane Imaris multi threadingBitplane Imaris multi threading
New Post
Bitplane Imaris multi threading
 

Concurrent Processing of Multiple Imaris windows  (run code including  ImarisXT in an un-conventional way to process ims file list) was accomplished via Matlab under the "SuperColoc" : A Matlab-based smart, self-rescuable and utility extensible job-queueing framework for robust and unattended batch processing of images using Imaris.

The number of windows could be specified from 1 to infinity, as long as the hardware could cope with,

For a pilot test, 12 Imaris windows runs fluently on PC with 64GB memory and E5-1620 8-core CPU. The optimal number of concurrent Imaris windows was depend on the memory, CPU, imaris file size and computational workload of processing an image.

On a single machine (personal PC, image workstation, HPC…), the maximum number of Imaris windows was limited to 512.

On distributed computing server, there will be no limiting of number of Imaris windows.

 Imaris files could even be concurrently processed in the Continental distributed computing server.

 
Copy Post URL
video snapshot at
https://www.youtube.com/watch?v=uzlIIhqi3As
 
Copy Post URL
There is still one last problem:

For each imaris window, after processing files one-by-one, the memory usage always increased and gradually use up the memory, and may be finally crushed(although I have not seen it. What I see is that memory usage keeps at ~100% but imaris still runs without problem). The reason may be that datasets or objects that was created by Matlab command but not loaded to surpass scene will hold place somewhere in memory even after imaris application opens the next file, unless the current imaris application quit. A reasonable property of Imarisapplication should be that once ImarisAplication.FileOpen() was executed it should clear all the memory related to previous file. However, this was not the case by now.

Another video at:
https://www.youtube.com/watch?v=8Al7JoZrYlQ
 
Copy Post URL
The virtual memory take space from C drive. Just now I see that C drive usable space drops from 82GB to 68GB.
My memory has 60GB, plus 60GB virtual memory.
The commit(GB) a while ago was 104/127, but now it truns to 136/148. So the system is always increasing the virtual memory to accomodate the memory need of imaris.
 
Copy Post URL
a new problem:

The first processing of 354 ims files. After processing 241 files, all the windows seems sleeping. Although memory usage is 106GB/165GB, the CPU dropped to 0. Althogh all imaris windows shows images, no activity appeared on imaris windows, and matlab also in normal waiting as usual. This last for 20 minutes untill I noticed them. I just click the imaris windows on the task bar, then these windows wake up one-by-one, and the processing resumed as normal. .So weired.

 
Copy Post URL
I have just enabled SuperColoc to detect the total commit memory before processing each file in case the total commit memory is close to upperlimit and automatically re-start imaris.exe. Also enabled the auto-click to activate all Imaris windows one-by-one in case that all these windows show no CPU usage.
 
Copy Post URL
 
Copy Post URL
SuperColoc also has an interface to ImarisXT so that any single ImarisXT step or a series of ImarisXT steps could be run at multiple imaris windows.
 
Copy Post URL

Four mode of automation:


Mode 1: Tandem, interrupted thread 
--Slowest, but most robust and stable. 
--Preferable on low-performance PC/laptop/low-level office computer.
        --Thread is monitored real-time to rescue any catastrophy in memory.
        -- RAM was released after each file processing. 





Mode 2: Tandem, streamed thread
--Faster than mode 1.
--Preferable on medium-performance PC/laptop.
        --Thread is monitored real-time to rescue any catastrophy in memory.
        -- RAM was challenged to near its upper limit then released, keep cycling untill task finished.



Mode 3: Parallel streamed threads
--customizable threads (≤ 512 threads/local cluster).
--Preferable on Workstation/HPC cluster.
        --All threads are monitored real-time to rescue any catastrophy in memory.
         -- RAM was challenged to near its upper limit then released, keep cycling untill task finished.



Mode 4: Distributed Computing Servers
--Multiplication of mode 3.
        --Rescue disabled
        -- -- RAM was challenged to near its upper limit then released, keep cycling untill task finished.

 
Copy Post URL
Theme of SuperColoc:

Once a guy could make sure what he decide to do on a single .ims file using Imaris, there is no need to hand-operate on Imaris software, no matter how many .ims files he may have or may generate later. There is even no need to generate .ims file from his microsopy raw images by hand. SuperColoc take over from the fresh raw image right off the microscopy, ending with figure and plot in your email.


Step 1:
Task Scripts editing

Each person plan his own task(s) immediately after his confocal image acquisition.
For each task, just answer/update several short answer questions (from a dozen to hundreds) in a text script indicating everything to do. One of the questions was specifically designed to run your ImarisXT(s).

Step 2:
Task Scripts feeding

Feed the folder containing all/unlimited task(s) from all persons in a lab/department to SuperColoc, then leave computer alone.

Step 3:
Data review

After each Task finished, an email will be send to respective person. On the same/following days, check email for data analysis results (plot/graph/figure...), running report and running log.

Go back to computer to check the detailed data (better when all tasks finished).

 
Copy Post URL
How robust is SuperColoc?

SuperColoc was initially designed under the premise that all the task it is processing containing all kinds of unexpected errors, either deliberate or inadvertent errors in task design. This ensure that when multiple tasks was scheduled and processing one-by-one, the error of a task will not halt SuperColoc or affect the processing of other tasks in list.
 
Copy Post URL

What’s catastrophy rescue in memory?

SuperColoc has divert the main body of data processing away from the Imaris window, so the robustness of data processing was astoundingly improved, especially for general computers. But even so, with intensive data processing via ImarisIceServer and Matlab, there is a chance that certain peculiar and rare memory error such as Byte Buffer Error will take place at late stage especially when there are lots of files to process and the computer is not well-performance. This kind of error will permanently deranged the memory block assignment regarding to ImarisIceServer and will never recover, unless the computer was completely shut down and COLD-re-started(even the so-called fast reboot or hot restart will not solve the problem). SuperColoc will restart computer at this circumstance and continue from exactly the imaris file where the task was aborted.

 
Copy Post URL
How to select the mode to run?
%(Q88)[---runStyle-------------]keep IGNORE as default intermittent threads, or configure the run style of streamed threads:

stream


[If use LOCAL cluster, the #, note # <= 512 upper limit, of expected Imaris threads to launch simultaneously locally. A few threds may abort once born, so will further trim to ACTUAL number. Then will try hard to check and make sure at least equal number of local workers on.]

[elseif use MDCS cluster, the total #, (note # must <= pre-configured NumWorkers on this MJS cluster profile, which is further <= Total workers your asset provided, which is equal to "total node/PC numbers" x "uniformly pre-configured workers per node/PC"), of expected number of workers to use as fraction(<= 100%) of this MJS cluster profile],

(Deprecated but please retain! LOCAL cluster effective only: If # > 1, the TIMEWINDOW in seconds for updating threads monitor in LATE phase after injecting the final file into imaris threads)

{Deprecated but please retain! LOCAL cluster effective only: If # > 1, in LATE phase, the number(##) of TIMEWINDOWSs above which I have to break since it is intolerable that threads may be frozen and file progression was not observed after a span of ## times, indicating that all remnant thread(s) potantially stucked at the end of file Queue. So kill the stuck for me.}.

@Maximum Commit Memory in MB allowed to break and resort to retryFailedList, which shoule be set to a little bit, e.g., 8% less than minimum of (RAM + configured SWAP.max) and (RAM + Empty space on SWAP drive)@

----If stream, NEVER use SmartSave/FixedSave since they are clocked bomb/killer of All ImarisThreads.

----If stream, should always use bfSurfsClear or/and AftSurfsClear since new .ims file injection dose not always clear previous objects.

----If stream and Hide Imaris, better not try to make ScreenShots since it have to annoyingly pop up when ScreenShot.

----If use MDSC cluster, also attach MDCS and $ProfileName$ and = "number of workers setting per node/PC" since probably most nodes will each make ALL their workers affiliated to MJS profile, especilally if no other guys beforehand submited their MJS so consumed some workers.>

Answer in Task scripts:
xxx.runStyle = 'IGNORE?--stream[1](20){30}@150000@MDCS$myMJS$<5>';
 
Copy Post URL
How to run your ImarisXT?

%(Q87)[---ImarisXT-------------]keep IGNORE, or input your assembly of ImarisXT names to run. The ImarisXT must be in search path of matlab.
xxx.ImarisXT = 'IGNORE?--{''XT_MJG_Surface_Surface_coloc'';''XT_MJG_SurfaceSurfaceContactArea'';}';
 
Copy Post URL
How to turn on catastrophy rescue?
%(Q86)[---BufferRescueMode-------]

keep IGNORE, or input the Mode to refresh memory once rare DateBufferBytes takeplace to rescue whole project.
1 indicate SchTasks, 2 indicate RunOnce.

Also attach the successively bufferfailed files [#] to trigger. Whatsoever, must ending with [#] so that if IGNORE BufferRescueMode, # indicate SuccessivelyFailedFiles upon which to notify MemoryFatigue to hibernate PC anytime.

To enable BufferRescueMode, also

(a)The post-WINDOWS8 "FAST STARTUP" must be disabled to make cold boot.

(b)The current instance of Matlab must be run as Admin mode(Not make your system as Admin(as most people run PC in User-mode like me, not Admin mode), but switch matlab to Admin mode in non-Admin environment), if not Admin, auto-triggered rescue will be locked at first step and halt your PC there forever

xxx.BufferRescueMode = 'IGNORE?--1[5]--2[5]';
 
Copy Post URL
How SuperColco convert your raw images to .ims files?

%(Q02)[---subAsk------------]Input Yes if your images are Grouped into sub-folders.
xxx.subAsk = 'Yes?';
%(Q03)[---imgFmt------------]Input the Format of your image with the dot, e.g. .TIF, .ims and so on.
xxx.imgFmt = '.ims';
%(Q05)[---unityCheck--------]Input Yes if EACH of your biological sample correspond to only ONE file. If Yes, skip the following questions until Q15.
xxx.unityCheck = 'Yes?';
%(Q06)[---RenameCheck-------]Input Yes if Channel-Index-Type such as _/-C0, _/-C1,or _/-C001... Already Exist in ALL images, instead of Channel-Name-Type such as GFP, DAPI...
xxx.RenameCheck = 'Yes?';
%(Q07)[---chName1X1CeInput--]Input FULL Assembly of Exact-Complete-Strings of CHANNEL NAME From ch=0 Onward if Channel- Name -Type; Else if Channel-Index-Type, input your choice of name with respect to Channel Index from ch=0 onward.
xxx.chName1X1CeInput = '{''w2CSU RFP'';''w1CSU GFP'';''DAPI'';}';
%(Q08)[---spaCer------------]Input the Spacer which act as seperator among the C, Z or T channels from your images, usually such as _ or -.
xxx.spaCer = '_';
%(Q09)[---Exis--------------]Input Yes if Channel-Name-Type AND such spacer Exist on left side of the Channel Name, else SuperColoc will insert one Spacer there.
xxx.Exis = 'Yes';
%(Q10)[---spaCerRplc--------]Input a NEW single character to Replace the unwanted Spacer which might appear unexpectedly on Left side (filename main body).
xxx.spaCerRplc = '-';
%(Q11)[---spaCerRplcFml-----]Input Matlab Expression defining which Spacer(s) on Left side (filename main body) to be replaced to NEW single character.
xxx.spaCerRplcFml = '1:(length(nich)-1)';
%(Q12)[---timeSqJudge-------]Input Yes if images are Time-sequence type (*t1.*,*t2.*,...)AND you Want to split timepoint for individual analysis instead, if need.
xxx.timeSqJudge = 'Yes?';
%(Q13)[---timeExtractL------]Input Matlab Expression of position of First character of Time Channel
xxx.timeExtractL = 'nich = strfind(newname,''_'');timeExtractL = nich(length(strfind(newname,''_''))-1)+1;';
%(Q14)[---timeExtractR------]Input Matlab Expression of position of Last character of Time Channel
xxx.timeExtractR = 'nich = strfind(newname,''_'');timeExtractR = nich(length(strfind(newname,''_''))-0)-1;';
 
Copy Post URL
Biological images are grouped into 3 levels:

Inferior: an image set was stored in multiple files, such as .tif of Metamorph.
Equivalent: an image set was stored in single file, such as .oir of Olympus FV3000.
Superior: a file is assembly of more than one image set, such as .lif of leica.

SuperColoc by now support Inferior and Equivalent, but not Superior because each Superior format is unique and should be treated individually.. If force SuperColoc to deal with Superior, perhaps only the first image set of the assembly will be converted.

 
Copy Post URL
All the other questions in between determines what to do on each ims file. This part is highly customized and extensible.
SuperColoc itself already has several highly developed steps:

Surface clear

Rename channel

Unify Voxel Size

Re-define VoxelSize

Channel mathmetics

Morphological operations highly customized: imerode/imdilate/imopen/imclose/imtophat/imbothat

Median Filter with optional median-filter based margin background subtraction

Gauss Background Subtraction Filter (appear multiple times)

Free defined Histogram Equalization with [currentMin,CurrentMax,newMin.newMax] and even Gamma corrected histogram projection, more customized than Imaris embedded, Super Fast! Code highly optimzed so that it take less than 3 seconds. The [currentMin,CurrentMax,newMin.newMax] could be dead number or percentile or dead dome voxels of ranked whole data block.

Gauss Blur Filter (appear multiple times)

Masking Surface Detect: This was the central part of procesing step of SuperColoc, The threshold could be dead or dataset-based formulized.

Finer processing of Masking Surface: culling to retain the largest 2 component.

Surface cropping dataset.

Masked Surface Detect

Surface-Surface Colocalization

Stats export

Highly customized stats analysis, plot, figure, csv...

Set channels display range, gamma, and alpha, RGB, and visibility.

Set visibility of all Surpass components individually.

Save file: four mode: 'FullSave?--FixSave?--SmartSave?--NoSave';


Snapshot

=======================
How to extend SuperColoc processing steps:

Just:

(1)insert your modulized code (something similar to ImarisXT but without the preamble and need minor and very very very simple modification) to a bottom function of SuperColoc. 

(2)optional: add(update) questions for argument in task script if you want to re-use the code in future with different parameter. 

(3)add two line: add/update a new field to an identity struct at  beginning and end of each newly inserted modulized code block, for SuperColoc to track, and also must accordingly update this new fields to identity struct definition template script.

That's all.

 
Copy Post URL

The process for file conversion to ims file at the beginning is also highly customized so that the number of paralled ongoing conversion of image set could be set from 1 to infinity. For a single conversion, the number of sub-thread could also be defined. So it is a multi-threading of the multi-threaded file convert.

 
Copy Post URL


Post URL:
casque beats pas cher casque beats by dre pas cher beats by dre pas cher beats by dre pas cher maillot foot pas cher maillot de foot pas cher maillot foot pas cher 2014 maillot de foot pas cher maillot foot pas cher maillot de foot pas cher maillot foot pas cher 2014 maillot de foot pas cher maillot foot pas cher 2014 maillot de foot pas chere maillot foot pas cher maillot de foot pas cher Ralph Lauren Homme Polo Manche Longue Ralph Lauren Polo Ralph Lauren Pas Cher Homme Polo Ralph Lauren Homme Pas Cher Ralph Lauren Soldes Ralph Lauren Big Pony Ralph Lauren Soldes Ralph Lauren Big Pony Ralph Lauren Soldes Ralph Lauren Big Pony Ralph Lauren Soldes Ralph Lauren Big Pony