4th August 2016
STOCK in APSIM Next Generation where are we at? - Neville & Eric
- Stock already running in feedlot in APSIMx
- Neville to implement new designed animal / stock defoliation interface. New interface before end of Jan. Andrew Moore to circulate.
- AgResearch to do testing from Jan - Jul. Publication by Jul 2017.
- Neville to get some management examples running this FY.
- Need to change REPORT to deal with lists of structures and arrays that change size during a simulation.
- Andrew is keen to put pastures into PMF. Andrew to circulate a discussion document.
IAT rewrite in APSIM Next Generation - Shaun
- IAT is a small holder model focused on herd management and growth and economics to suit small holder farms in south east asia and other developing countries. Shaun is rewriting IAT from scratch using APSIM Next Generation engine and some models. It runs on a monthly timestep and imports APSIM (and other model) outputs. The short term goal is to reproduce the current IAT functionality e.g. keep the IAT stock model and ipport pasture growth numbers. In the longterm, the plan is to turn it into a daily timestep (IAT models will still monthly internally), have it run with other APSIM models, have the ability to use the APSIM stock and pasture model and have feedbacks between all models i.e. closer to a normal APSIM model.
- There was some discussion on wanting to rename the folder that this work is being committed into. Reference Panel don't link the name 'wholefarm' as the folder name. This will be discussed at the Reference Panel.
- Aim is to reproduce IAT functionality by end of 2016/17 FY.
GTK version of the user interface - Eric
- Goal is for APSIMx (model and user interface) to be usably across Windows, LINUX and Mac. MONO wasn't working well on all platforms and so Eric has been working on a prototype of APSIMx using GTK# instead of Windows Forms. This is now working on all 3 platforms.
- Eric gave a live demo showing the new user interface running well on Windows.
- A downside is that user interface developers will have to use GLADE instead of Windows Forms designer.
SWIM - Justin
The conversion from Peter Ross' FORTRAN to C# in APSIM x is almost complete with many unit tests now proving that it is giving the correct results. Still needs to be integrated into user interface. Also needs test simulations before going to Reference Panel approval.
14th November 2013 APSIM Developers VC meeting
Attendance: Dean Holzworth, Justin Fainges, Shaun Verrall, Peter deVoil, Hamish Brown, Neville Herrmann, Eric Zurcher, Val Snow.
- We reviewed the progress on ApsimX to date since the August ApsimX workshop. The core models (SoilWater, SoilNitrogen, SurfaceOM) have been ported to ApsimX. Hamish is working on Plant running in ApsimX (should be completed this week).
- We stressed the importance of testing these models with observed data where possible. Very important to get to the point of being able to release a first version of ApsimX that we have confidence in the outputs. We would like to have a first version by April 2014.
- Action: Shaun to bring across the SoilWater process tests into ApsimX ASAP (by end of January?)
- Action: Val to coordinate the addition of some tests for SoilNitrogen and MicroClimate.
- Action: Dean to work on a .apsim to a .apsimx converter.
- Action: Dean to find someone to port AgPasture from C# APSIM to C# APSIMX.
- Action: Need to get old style SLURP (tell it everything) running in Plant. Val needs this for SoilNitrogen tests.
- Action: Justin to work out technology solutions for running a virtual ApsimX "training / usage" workshop.
- Action: Dean to find a date for a virtual ApsimX "training / usage" workshop in December. Depending on how this goes we might have a face to face workshop early next year - end Feb at the earliest.
- We also discussed the idea of doing an APSIM release (7.6) early in the new year.
- Action: Dean to go through the task list, looking for important defects that must be fixed, and assigning them to people.
19th - 20th August 2013 ApsimX design workshop
Attendance: Dean Holzworth, Neil Huth, Justin Fainges, Shaun Verrall, Peter deVoil, Andrew Moore, Neville Herrmann, Eric Zurcher,
Chris Murphy, Greg McLean, Al Doherty, Hamish Brown, Jo Sharp,Rogerio Cichota.
Apologies: Val Snow
Monday 19th August
Overview and objectives of ApsimX (ApsimX workshop.pptx)
The model framework
The model framework (continued)
The user interface
The user interface (continued)
Tuesday 20th August
Building, testing and deployment
First version of ApsimX (what goes in it)
Action plan (who does what)
The ApsimX design document that was refined over the two days can be found here: ApsimX Design.docx.
Some notes from the two days (thank you Andrew):
- All “zones” are “systems”, but not all “systems” are “zones”. Example: it will be a good design to run Stock as a “system” that can contain a parasite model (WormWorld). Potential solution: revert to “system” concept, plus have a “Spatial” interface that must exist at some level in a validly defined simulation
- Keeping the Simulation class to the specific task of simulation (i.e. integrating a synamics model over time) is highly desirable. Therefore, e.g., running POAMA-style tactical analyses should be assigned to a controller class that invokes one or more Simulations
- The claim in the design document “By the time [OnInitialised()] is called, ApsimX will have satisfied all link references” is untested. Agree that the INIT1/INIT2 approach currently in use is only a partial solution; but I suspect that the correct approach is to iterate link resolution & initialisation steps until all models report that both tasks are complete. Models that create variables dynamically (e.g. manager components) are the difficult case
- Scoping rules: the case where a model in a higher-level zone requests a variable that can be found in a lower-level zone needs to be covered
- Scoping rules: my recollection from the CMP design process is that a large set of scoping rules can be defined with quite a simple algorithm description (2-3 integer parameters). Should we generalize now? Classic waterfall/agile tradeoff…
- Will the rules scoping of events match those for scoping of variables?
- Data updating (“reset” functionality): the current design has a single definition of “settable” at both design and runtime. There is a need to separate these (equivalent concept to “initialisable” vs “writable” in the CMP)
- The Model class/interface needs to implement a Parent method so that querying for variables & events over the simulation tree can be implemented. (This process should really be implemented generically in the Model class…) Example case: resource competition arbitrators
- Need to preserve arbitrary timestep length
- Ordering of computations within each time step: the start/middle/end events will prove inadequate. Happy to see an event-propagation approach (“tick” followed by notification events from models) pursued
- User interface: will a manager model in the UI be able to report its internally defined variables to a reporting model to allow users to select them?
- All models in a single exe: a “must” approach to this prevents a user from attaching their own model to an APSIM simulation. This would be undesirable from a relationship-management point of view; the case where people move toward participating in APSIM gradually is valuable (ORYZA or OZCOT as examples?)
The action plan from the two days
Work for next two months:
- Repository Git vs Svn Pete
- Continuous integration tool selection (Cruise Control?) Pete
- Testing / stats / R framework. Justin & Paul.
- Write a sequence diagram for a complete simulation and interfaces Eric
- Write translator: .apsim to .apsimx Dean
- Decide on User Interface display technology GTK#, Mono Develop? Dean
- VC October / November to assess progress on the above items.
- Zone (add tests) Neville
- Tests for clock – check for regional settings Neville
- Manager Dean
- Rework code to run in ApsimX Shaun
- Tests/Validation Justin with Neil and Hamish
- Rework code to run in ApsimX Shaun
- Tests/Validation Rogerio & Jo
- Surface organic matter
- Rework code to run in ApsimX Eric
- Tests/Validation (validation with sugar, wheat) Jo + Neil
- Irrigation, fertiliser, sysbal? Justin
- Report Dean
- Rework code Eric
- Tests Neil
- Summary file? How? Text vs binary? Both. Dean
- Operations model Peter
- Weather model Neville
- add tests,
- add daylength,
- adding hourly temperatures,
- calc tav/amp from file or use global gridded lookup (Neil knows what to do).
- User interface Dean
- Links (shortcuts) in GUI – how?
- Internationalisation issues in GUI.
- No spaces in model names. GUI to enforce.
- How is a soil represented in GUI and model. (single soil, child models for soilwater, soil nitrogen)
- Auto updater from web
- Below ground resource arbitration. Define interface and implement dummy model. Will modify Plant2 to use interface. No need for a real arbitration model yet. Val & Hamish
- Potato, do this before wheat. Feb 2014 decision point on whether other crops can be done. Hamish.
- Wheat Neil Mar-June
- Sugarcane (optional) Shaun
- OilPalm (optional)
- agpasture (optional)
- Soil temperature (optional)
- Remove from soil nitrogen
- Add tests.
- What do we do about models not OK’d by Reference Panel? Devise an exclusion list? Something better?
- Solute (optional)
18th March 2013
Attendance: Val Snow, Edmar Teixeira, Rob Zyskowski, Frank Li, Rogerio Cichota, Paul Smale, Neville Herrmann, Eric Zurcher, Dean Holzworth, Justin Fainges, Peter deVoil, Jo Sharp, Shaun Verrall
- The meeting discussed the pending APSIM release 7.5 due sometime in the first couple of weeks in April. The implication is that all work for the release needs to be completed before the end of March. The APSIM development team will then have a couple of weeks in April to test the suitability of the build for release. A testing day will be organised early April. During this testing phase, only major faults will be repaired. No new functionality will be permitted.
- The meeting then reviewed all assigned tasks from the APSIM bug tracking system to determine whether they were going to be completed by the end of March. The consensus was that those present would close tasks that have already been completed and either implement the remaining ones or remove the medium priority from those that won't be done in time.
- Val Snow was asked whether the AgResearch way of running SWIM from the GUI would continue in APSIM 7.5. This cannot be done until we can have a nitrifiction inhibitor and a tracer in the SwimSoluteParameters component - a user interface issue (Task 1438: http://www.apsim.info/BugTracker/edit_bug.aspx?id=1438). Dean might look into getting this done if time permits.
- Regional settings in APSIM and the user interface are still an issue. It was noted that this is difficult to fix completely, particularly for the different number formats from different cultures. Justin and Dean will explore this further before the release. It would be nice to resolve this at least partially before the training workshop in Brazil in April.
- There was also a discussion about no new .NET components (SoilWater, SoilNitrogen, SurfaceOrganicMatter, Wheat and Sugar) to be released in APSIM 7.5. The development team ran out of time to have these incorporated into the new release.
- Jo Sharp is visiting Toowoomba this week to begin developing a way to improve how the build machine (Bob) checks for change in the test suite. The idea is to develop a smarter method that is capable of comparing validations against observed data and sensibility tests at the probability level. Current Bob compares .out and .sum files at a binary level.
24th June 2011
switching over to CSDotNetComponentInterface
o Do it incrementally – SoilNitrogen.dll then SurfaceOrganicMatter.
o Might be a bit slower – 10% slower – Neville / Eric to document the speed penalty.
o Dean to merge his DotNetComponentInterface changes into CSDotNetComponentInteface next week.
o In the short term use Plant2.sln. Eventually switch to using CSPlant2.sln.
o Neville has got DYMEX working nicely in APSIM. He has forwarded to Jeremy Whish for testing his nematode model.
o Is available for others.
o Not for inclusion in an APSIM release yet due to IP restrictions.
o Neville isn’t sure if purchased DYMEX dll’s can be used with APSIM although if DYMEX is purchased then legally it is probably ok for Neville to distribute his version of DYMEX for use in APSIM.
o Appears to be working in APSIM.
o David Johnson is tinkering with the APSIM user Interface. He still has to get the validation datasets moved over from con/par files and get them working in the user interface.
o John expects the end of the year for a version to go into APSIM.
question about when is the next release due and what are the goals for then?
o End of the year. The only real demand driving this is the need to fix the soil mashing problem from APSIM 7.3.
o Neville will let David Johnson know about the release so that he has a deadline to work towards.
in relation to the last point, is it feasible to finish the gfortran port for the release?
o Would be nice to have gfortran in release - Eric
o Good to have SWIM, the new Manager module and SoilTemp2 in the release.
o Ben to commit a tool for checking diffs in output files.
o Dean to ask RP about switching to the .NET version of SoilNitrogen, SurfaceOrganicMatter, SoilWater.
Speeding Up Builds following commit.
Process for keeping the build clean.
o No consensus was reached about improving the commit / build / run cycle.
o Pete / Dean to look at implementing Ant for compiles and simulation runs, complete with dependencies. That way only those simulations affected by a commit will need to be run.
Plant2 progress update.
o Hamish reported on his visit to Toowoomba, tiding up Plant2, arbitration, cohorting leaf.
o Hamish is to write a MODSIM paper on Potato2 in July.
o Neil to implement senesced leaf detachment process in Plant2 soon.
We need to discuss a fix for the issue Val et al is having with the binaries being sandboxed because they were downloaded from the net. We've got a work around in that you can manually unblock the ZIP before extracting it, but we may need a more permanent solution.
o It’s a windows security issue.
o No longer a big issue for Val now that she knows the workaround. Probably no need to change anything in APSIM or on Bob at this point.
Invalid installation files
o Installations should be ok now. Dean has fixed them recently.
Soil Nitrogen N2O module – how should we deal with this
o Frank Li has been working on different algorithms for N20 in the new SoilNitrogen C# version. Should he commit them even though SoilNitrogen.dll hasn’t got any tests? Probably not. It would be better to wait for the switch to SoilNitrogen.
o Eric to port Micromet to C# and then we’ll convert to VB for Val.
Net CDF or SqLite for a alternate REPORT module for APSIM?
o Net CDF is a simple binary format but lacks the query ability of databases. Dean will probably go for a database solution for this reason.
o Dean to investigate the solution in AusFarm which writes to a database.
22nd September 2010
Bob the build machine
Dean reported that he was working on revamping Bob with an aim to producing a more stable build machine. Bob is now a Windows 7 64bit computer with a revamped build system that is capable of asynchronous compiles. Windows 7 64 bit also solves some of the slow down issues we were seeing with builds. Dean expects that Bob will be back up sometime within the next week.
Plant 2 workshop
Dean reported that another Plant2 3 day workshop is being planned to further the development of several plant models with Plant2. The most likely week is the first week of November in Brisbane. People x Models:
Heidi and Shaun (Sugar cane)
Hamish (potatoes and barley if time allows)
Edmar (lucerne and maize if time allows)
Rob (kale and turnip if time allows)
Alex (Annual ryegrass)
Frank LI (AgPasture?)
Fast APSIM Prototype
Dean reported that a workshop / meeting is to be planned to discuss developing a prototype of a fast running APSIM. This to be held in Canberra. Dean, Peter deVoil, Neville, Eric and Ben to attend this.
Val also suggested adding a day onto the end to talk about developing a "simulation exploder" tool (to quickly create multiple simulations). Val is interested in coming along to this.
More details to follow on this workshop.
Exported function calling conventions on various platforms
Neville & Eric reported on their success on getting APSIM running on Linux and the fun they've had! They had a question about why APSIM is using "C" style exports on Linux rather than the normal "StdCall". This is a question for Peter deVoil (who wasn't in attendance).
Progress with Fortran compilers
Neville & Eric reported that they are able to compile APSIM successfully on Linux and Windows using GFortran. Dean suggested that it is a priority to replace the aging Lahey compiler currently in use on Windows platforms.
Progress with APSIM/AusFarm mixed simulations on Linux and Windows
Similar to the above, Eric and Neville have managed to get APSIM and Ausfarm mixed simulations running on Linux; a great achievement!
Ideas on SoilWat & Erosion future development
Shaun reported on progress in getting a .NET version of SoilWat running. He's managed to sort out interface (variable) issues and has a empty shell of SoilWat running. Similarly Eric is beginning to work on a port of SoilN to .NET.