Skip to content. | Skip to navigation

Personal tools

Sections
You are here: Home / Concepts / Task Type System

TaskType System for Evolvix

The following notes were composed on the occasion of the 2017 HTCondorWeek, where a presentation was given on how Evolvix and HTCondor might combine in the most user-friendly and flexible way. This is best done by introducing a type system for tasks.

The talk at HTCondorWeek explored how bridges might be built between Evolvix and HTCondor. A case was presented, for why most biologists will probably never want to know about HTCondor, even if they end up using it. By presenting some draft interfaces the question was raised, how a type system for globally distributed computing tasks might be integrated into a general-purpose programing language. Below are relevant links for linking into the FeedbackFlow, the latest slides as PDF, an informal summary and some additional links.


 

The most up to date PDF with the latest design-updates is available at:

     http://evolvix.org/concept/tasktype/task-type-system-talk-MMv2/view

 

Please provide Feedback through the FeedbackFlow established at

     http://evolvix.org/FF





Title: Evolvix and HTCondor - What would it take to integrate a type system for computing tasks into a general-purpose programing language?

by Laurence Loewe

presented at 2017-05-05 at HTCondorWeek at UW-Madison


Abstract: Modern programming languages in scientific computing may be used for extremely diverse computing tasks. For example, predicted CPU time complexities (at scheduling time) for simulations could range from predicted milliseconds to centuries as observed in the evolution@home global computing project within a single parameter sensitivity analysis. Obviously, the fastest simulations are best computed on a single CPU (no scheduling overheads), while the longest running simulations would not be worth scheduling on any contemporary computer (waiting for a CPU of tomorrow will probably still get results much faster). Between these extremes exists a wide range of simulation tasks that are worth parallelizing for appropriate HTCondor compute pools. To do so efficiently, it would be ideal, if users would only have to declare to the compiler of the simulation language, which compute resources they wish to use, what their limitations are, and how to access them (if not local). The compiler would then automatically decide, whether a given task is best computed locally, bundled with others, flocked to a certain HTCondor pool, or best flagged to the users as "probably too complex to compute".

A draft design will be presented for a type system that categorizes computational tasks according to their predicted or observed complexity as measured by various criteria. This system aims to introduce a separation of concerns between code defining simulations of scientific interest and code handling the practicalities of where and whether to compute them. The language constructs aim to simplify the coding complexity of many (but not all) diverse, recurrent simulation scenarios for programming biologists. This can work if the compiler producing the executable for the simulations can also compose DAG workflows and schedule HTCondor jobs on compute resources with known limits. While approaches presented target the Evolvix modeling language, they could in principle be used in many other languages as well.

This is work in progress intended to invite the discussion and feedback necessary for defining the requirements of an efficient and user-friendly implementation.

 


 

Please provide Feedback through the FeedbackFlow established at

     http://evolvix.org/FF

 

The most up to date PDF with the latest design-updates is available at:

     http://evolvix.org/concept/tasktype/task-type-system-talk-MMv2/view

 

HistoryHeap

Previous versions of this design PDF:

     http://evolvix.org/concept/tasktype/MMv2 (PDF direct, 5.6MB)

     http://evolvix.org/concept/tasktype/task-type-system-talk-MMv2/view

 

The talk in its original HTCondorWeek context:

     https://research.cs.wisc.edu/htcondor/HTCondorWeek2017/presentations.html