This post was last updated on 9. of May 08.


Latest update is here


A new science application called \"optimizer\" will be launched from Monday 17.September onwards, or shortly there after. Watch this thread for news, it will be announced when we start.

At first, it will be run as a test application, meaning that only users who have \"run test applications\" and \"run optimizer application\" checked in their account settings (under malariacontrol.net preferences) will get work.



In addition, only windows hosts will get work.


Work units will take from 5 min up to one hour, depending on the model parameters. No checkpointing is done, and progress will not be indicated, so wait at least for an hour before thinking the work unit could be stuck..
Calculation is done by a java program, contained within the standard boinc-\"wrapper\" application. You need to have java installed.. if not, you will be prompted to do so.
Deadlines: in the testing phase 48h, after that three days.

The name \"optimizer\" for this application was chosen because the server side components are essentially a \"general use\" optimization framework to be used by scientists in our group to work on more specific questions. E.g. to fit simpler models for which the \"big\" malaria model would just not be what you want. The insights from those calculations will help us to improve the main malariacontrol application in the future.


On the science of the project:


To make quantitative predictions of malaria transmission, it is very important to know how long an infection lasts in an infected human. Because the longer it lasts, the more mosquitoes can get infected, the more infected mosquitoes you have, the more humans are being infected etc, etc, etc..
It may at first seem very straightforward to measure this: you just look when somebody gets infected, and then you keep taking blood samples until that person is not infected anymore.
Unfortunately, you only have a chance of about 50% percent to detect an infection, given that it is there. So you already have a problem: you don\'t know when the infection started, and you don\'t know exactly when it ended.
In addition: In areas of high malaria transmission people are very often infected with up to ten or more infections simultaneously... so you never know if what you\'re seeing is still the same infection or a new one..
Recently some work at our institute has used new dna-based methods (which allow distinction of different infections), together with a mathematical approach, to estimate the average duration of an untreated p. falciparum infection.


see Sama etal. 2006
(sorry, only the abstract is freely available to the public)


So far so good, this was an important step forward. The problem that remains is: how are the durations distributed? In other words: do all of the infections last exactly 200 days and then all of them stop? Or does an infection have a constant probability to disappear, which remains constant no matter how old the infection is? Probably none of the two is true, but we need to describe the shape of that distribution of durations somehow, in order to make sensible predictions.


for more on that, see Sama etal.2006b


That\'s almost where we want to go, except for one thing: the above paper measures the distribution in people living in the US who had never experienced malaria before. They were infected on purpose, to cure their syphilis (the method of choice at that time..) We don\'t know what the picture looks like in people living in areas of high transmission, with multiple infections at a time and after decades of being constantly infected...

Attempts to find a mathematical solution to this problem did not work out.. the equations become unsolvable. But there is a way out: instead of using equations, we can use individual based simulations, that means we simulate every single infection in a computer program, and see what parameters can best produce the data we have. The big drawback there is, this just takes too long to calculate on a single computer.
That\'s what we need you guys and girls for, and thanks a lot for making this possible!!

P.S.: Something about the data collection mentioned above, to prevent misunderstandings: There are strict ethical guidelines on how one is allowed to obtain such data. Since most malaria infections in high transmission areas don\'t cause any symptoms, being infected with malaria doesn\'t mean you are sick (because of acquired immunity). People who did have symptoms were of course given treatment.
____________
Michael

Will this only be an intermediate application to further improve the main app or will it stay as a stand alone application?

Thanks

Neat!
Is this the first time a java app has been used in the BOINC framework (across all projects, not just within MCDN)?

• There was a discussion group at the recent BOINC workshop in Geneva, mainly on how to ensure the presence of a virtual machine (JRE) on a client.
  
• At the same workshop, members of the SZTAKI group presented some preliminary work on distributing a JRE together with the science application
  
• There\'s an uppercase sample application in the BOINC SVN repsitory that demonstrates how to call the BOINC-API from Java.
  

As Michael said, we for now assume a Java runtime on the client, and use the wrapper approach.
Nick

____________
Nicolas Maire
Swiss Tropical and Public Health Institute
http://www.swisstph.ch

hi all,
we have now started sending out workunits of the new application. their names start with opt_ ... We had to resolve a problem with multicore processors first.. which we did.. then there was/is a second problem, which we have not yet been able to resolve:
- for every workunit, a temporary file temp<number>.jar is deposited in your c:\\Documents and Settings\\<username>\\local settings\\temp\\ directory (or whatever your temp directory for applications is - it may depend on your windows configurations).
The file is only 3kb big, and will be deleted on next reboot. For most people this should not be a problem, since we are currently only sending work to hosts with at least 750Mb disk space free.. that means you can crunch 250000 workunits before your disk gets full, and if you have one reboot before that, you\'ll start again at zero.. still, we are not quiet happy with this, please send us your comments, if this is a problem for you :)
Otherwise, maybe there are people out there who have experience with Jsmooth for launching java apps. Have you come across this problem and found a solution for it?
thanks
Michael
____________
Michael

This sounds like a very interesting approach, but I am curious if you would be willing to provide more details regarding the unsolvable equations. Given the MLE framework in the modeling of the article cited above, the individual simulations make sense. I am curious, however, whether you considered Bayesian approaches to the problem since a set-up with strong priors would seem to make sense (at least given what I was able to get from a cursory skim of the article and Metroplis-Hastings approaches to Bayesian MCMC models).

more than 30 PC run since more than 6 month. I hope you don\'t make more of this \"delete after next reboot\" things with bigger temps.
I really don\'t want that this pcs run out of hdd space ...
I hope you don\'t try to flood my hdd with this ;) Why should i reboot a pc if it can run without downtime :)

I\'m getting a quite few going into computational error due to max cpu time being exceeded......

Hi,

one of my 2 hosts is running optimizer application on this wu since some hours, but the CPU time counter is not progressing any single second. Also, in task manager, the total CPU time devoted to the optimizer process is 0.

It\'s a 3,00 GHz Xeon Server, so it has 2 cores and both are running BOINC processes concurrently, but, with other BOINC projects, this never led to one of the processes getting stuck.

Please tell me if I should abort this WU.

darkpella

I will post it here because no answer yet
http://www.malariacontrol.net/forum_thread.php?id=524

hi all,
the bug with the temporary jar files is now fixed with version 1.19.
Further, we had some problems on the server side, which should also be solved now.

Because of those problems (one server side component would crash every 10 hours or so), we have not been able to fully asses how long we would need to run the models we have now, in order to fit them all. But I have to say, I\'m quiet impressed by the computing power you are generating only within the group of people who are crunching testing applications!
From what I have seen up to now, this is far more than what I expected.
We agreed that in order to have the application leave the testing status, we would definitely need to bundle a java virtual machine with it - as part of the application. Otherwise this would presumably lead to a lot of confusion, especially among crunchers who don\'t have java installed, don\'t want to install it, don\'t know what java is, don\'t know how to opt out, don\'t know that this forum exists (or maybe are not so aware of it)..
apart from that, we find that its really hard to track errors due to the very heterogeneous software environment.. we can never be sure that its not a certain java version, or a damaged java installation, custom-made stripped jvm or whatever.. which is causing them.
So we decided to leave it as a testing application for the moment, because we seem to have enough computing power to get the results, at least for now, and that we might as well improve the science part of the application instead..

We do get errors, but not too many.About three to five machines are responsible for about ninety percent of them, so I strongly suspect some misconfiguration there. Either reinstall java, or if it doesn\'t help and you keep getting errors, just opt out of crunching the optimizer app.
Of course you are also welcome to post your problem in this forum, maybe we can find a solution.


____________
Michael

From what I have seen up to now, this is far more than what I expected.

hi all,
I just launched optimizer version 1.25 . It contains a bundled java runtime environment, so if everything goes well, you can run the optimizer app now without having java installed. That should pave the way to leave the \"testing\" status sooner or later.
regards
Michael
____________
Michael

Dear all,
the new application version of \"optimizer\" runs so well (ca. 1% error rate) that we decided to leave the testing state by mid this week. From Wednesday 21.Nov on we will start sending out workunits to everybody, not only the testing volunteers.

For more information about what the application is doing, see here

The java runtime environment is now being distributed as part of the application, and thus you will not need to have java installed in order to run \"optimizer\" workunits.


If you do not want to cruch workunits from this application, proceed as follows:

go to
Your Account -> malariacontrol.net preferences

set \"Run optimizer application\" to \"No\"

note that once the application has left the testing state, the setting \"run test applications\" will have no effect on whether you receive workunits from it anymore (though it does for the remaining testing applications).

thank you for staying with us
Michael

____________
Michael

Dear all,
the new application version of \"optimizer\" runs so well (ca. 1% error rate) that we decided to leave the testing state by mid this week. From Wednesday 21.Nov on we will start sending out workunits to everybody, not only the testing volunteers.

The java runtime environment is now being distributed as part of the application, and thus you will not need to have java installed in order to run \"optimizer\" workunits.

thank you for staying with us
Michael

What platforms are supported, is it still only Windows XP ?

I tried to run on some Windows 98 hosts with not so good luck.
They either showed client compute error
or as Done with 0.00 completeion time and 0.00 credit but credit is still pending. About 3-4 out of 12 may have completed this way.

I vaguely remember there was an issue with the wrapper approach, because win98 is not fully compatible with the windows api.. create_process() seemed to be a problem.. since we have taken the newest wrapper version of boinc, and we do get valid results from hosts with win98, I assume this issue has been resolved in the new wrapper versions - so no, it should be compatible also with win98.
The fact that some workunits terminate very quickly is because they realize early that the result is not going to be meaninful and terminate.. Unfortunately we can only do this at runtime, once the workunits are sent out.. and a wide range of durations is normal with this application, since it\'s a lot about exploring new types of models instead of calculating established ones.
Anyway, that\'s a personal opinion, but if I had old computers I think I wouldn\'t let it run on win98, I would take windows xp and tune it to the max (there\'s a lot of tutorials on the web), switch off all the things you don\'t need to run them as boinc client. i once did this with a 400Mhz laptop, and it actually worked ok, not slower than with something like win98, but much more reliable.. but as I said, that\'s my personal taste :)
cheers
Michael

What platforms are supported, is it still only Windows XP ?

I tried to run on some Windows 98 hosts with not so good luck.
They either showed client compute error
or as Done with 0.00 completeion time and 0.00 credit but credit is still pending. About 3-4 out of 12 may have completed this way.

Hi Keith,
I vaguely remember there was an issue with the wrapper approach, because win98 is not fully compatible with the windows api.. create_process() seemed to be a problem.. since we have taken the newest wrapper version of boinc, and we do get valid results from hosts with win98, I assume this issue has been resolved in the new wrapper versions - so no, it should be compatible also with win98.
The fact that some workunits terminate very quickly is because they realize early that the result is not going to be meaninful and terminate.. Unfortunately we can only do this at runtime, once the workunits are sent out.. and a wide range of durations is normal with this application, since it\'s a lot about exploring new types of models instead of calculating established ones.
Anyway, that\'s a personal opinion, but if I had old computers I think I wouldn\'t let it run on win98, I would take windows xp and tune it to the max (there\'s a lot of tutorials on the web), switch off all the things you don\'t need to run them as boinc client. i once did this with a 400Mhz laptop, and it actually worked ok, not slower than with something like win98, but much more reliable.. but as I said, that\'s my personal taste :)
cheers
Michael

Can you expalin why some hosts are showing ZERO time and claimed credit. Its not just Windows98. There is another thread started in which this seems to be an issue.

It appears to us that the hosts claiming zero time are all marked valid, but the effect it has is it lowers the claimed credit and possibly the cpu claiming zero is getting too much or cheated out, but there is no way to tell without knowing the actual time.

I think the users would like this issue addressed.

hi all,
after major refactoring, fixing, beautifying, adding stuff etc.. of the optimizer application, we are coming back online.. since the application has changed quiet a bit we will put it back to testing status for some time (at least today, let\'s see how it goes).. Just to make sure we didn\'t introduce some horrible bugs. Once a sufficient number of workunits has come back and everything looks ok, we will leave testing status and send work to everybody who has checked the \"run optimizer app\" checkbox in the account -> malariacontrol.net preferences. As before, you will be able to opt out of getting such workunits, by changing your account settings.
looking forward to new results (am very excited :) and thanks again for crunching
have a nice weekend
Michael
____________
Michael

So much for that then. 7 hours of No work from project, there was work but not for the applications you have allowed. I\'ve turned the Malariacontrol simulation back on and got fed from that immediately.

Will try again later this weekend.

LOL...yeah I even allowed test applications, but nada. It\'ll be interesting to see when the first wu\'s come through.

Live long and BOINC!

I\'m sorry guys, I closed the tap yesterday night, panicking a bit before going for a beer, because I had the impression something was wrong: One of my workunits finished three times but \"without a finished file\", as it said.. Did this happen to anybody else? Please check in your messages whether you keep on crunching the same workunit or not.. Migth be just a problem with my computer, though.. ok, and maybe I have to send out some more workunits in order to be able to ask someone :)thx

Please check in your messages whether you keep on crunching the same workunit or not..

That might be a problem with your BOINC version as well. I\'m alpha testing 6.1.8 just to see if I can get it to do those things, but that\'s difficult if all I get is the No work from project message. :-(

But I\'ll go back to waiting for an Optimizer.

I guess I should go abort those two opt WU\'s that have been running for seven hours then...

How about the one that says estimated time to completion over 35 hours?

update:

- new set of wu\'s is out
- make sure you have optimizer version 1.29 (should happen automatically). if not, abort them and do \"reset project\".
- estimated time to completion is still too high for the first few wu\'s.. in order not to shock the clients too much which were crunching some of the wu\'s before.. ignore estimated time at first.The second row of wu\'s has correct time indicated.
- maximum duration is 2h, anything that runs (much) longer is likely to be flawed, report please..
- errors due to \"time limit exceeded\" will not happen anymore
- pray for good success to whoever you wish :)

michael
____________
Michael

CreateProcess() failed - A device attached to the system is not functioning. (0x1f)
</message>
]]>

Validate state Invalid

FYI

as from now, the maximum duration of opt_ workunits is set to 3h. We want to test this, since we came to realize that for some specific runs, calculating an hour more would provide crucial information and speed up the fitting process a lot. At the same time we have adjusted the (server side) parameters as well, in order to make the fitting process more efficient.
This should mean, that those 3hour wu\'s just appear in the beginning, and should quickly become very rare (as the server side algorithm is now also more efficient and will after some learning time steer away from creating such wu\'s in the first place..)
Just don\'t kill them when 2h are over, as it was before.
thx
____________
Michael

Test of new optimizer app version

The \"optimizer\" application will re-enter testing status for a few days,
after some major changes to the application code. This means that during
this period, only users who allow to receive work for test applications in
there project preferences will get work for this application. Once the
testing completes successfully, we will change the status back.
This is expected to happen from next week.
____________
Michael

On one computer I got this message:
5/9/2008 10:17:46 AM|malariacontrol.net|Message from server: Estimation of parameters of infection dynamics (no progress bar, max 3h) needs 361.78MB more disk space. You currently have 353.48 MB available and it needs 715.26 MB.

This is wrong as I have 2+GB Free for BOINC, 4.56GB allocated and 2.17GB in use. This may be a client bug, but...

On another computer with about same disk usage, it got work, it had only 1.90GB in use so it had a little more room. And has 5 tasks but malariacontrol is only taking up 107MB of disk space for those 5 and any other previously downloaded application files.

So why does the application say it needs so much more ?
How much space does the application really require ?