Can I trust CLEM?

This is the most important question averyone should ask before deciding whether to use a model and goes to the very nature of using someone else's representation of a complex system with potentially limited engagement with those who developed the model. Holzworth et al (2021) in their MODSIM 2021 presentation (see abstract and presentation) suggest the answer to the question posed as the tile of this section is definitely not to trust any model, even one you are very familiar with, and assume all results are suspect until proven otherwise. We recommend you approach any model use with a healthy, sceptical attitude as this will benefit your understanding of the system, the model, and accepting the predictions. Trust in a model will arise from considering the questions outlined by Holzworth et al (2021) and we will address these from a CLEM development perspective to help answer the question of whether this model is suitable for you. We also realise there can be some strong sales pitches for the use of commercial models or models that need to prove uptake to maintain funding. In this section we believe an open and honest approach is the best to help answer this critical question.

This question relates to how the model has been compared with available measured data and whether this data is representative of your particular simulation requirements. This ensures the model process accurately represent the real world and your requirements are not beyond the tested boundaries.

While many aspects of CLEM are can be broken down to relatively simple resource accounting with all transactions reported, or tools to determine timing or filtering, or even fairly uncomplicated processes in activities, there are a number of sub-models such as the ruminant energy relations model (see Grow ruminants) that is based on a large number of published relationships. These models have been tested either in CLEM or in the precursor models. We have some tests available in the Validation folder of APSIM such as comparing feed quality and animal growth with feeding trials. CLEM also relies on the production output of other models or supplied data which you will also need to trust and approve.

The pasture sub-model is a hybrid model using other primary production input data, and your own detachment and quality decay parameters to represent the quality of a mixed age sward of pasture for grazing. We recommend you are fully satisfied with your pasture dynamics by interrogating the pasture outputs before accepting this section of CLEM if relevant to your simulation.

Update: The new Grow ruminants (Protein and Fat) ruminant growth model has been fully tested and validated against four published feed lot trials. This work is currently being prepared for publication. This work is also based on the relationships of the AusFarm model provided by Herrmann et al. (2024) Integrating animals, pasture, and crops within AusFarm for modelling mixed farming.

With simulations models such as whole-farm models growing ever more complex, and integrating a range of management options operating in a variable environment, it may not be possible to obtain measured data with which to compare the model output. This also relates to the difficulty in parameterising the simulation model to exactly represent an example farm at a particular time. For this reason, we suggest CLEM is used as a scenario exploring model and you even take advantage of stochastic processes to develop the range of modelled data to compare with known measures.

Sensibility tests use expert opinion to validate the model outcomes by asking the question "Do these results look ok and make sense?" This should be an ongoing exercise as you start using the model and exploring your results, or when results are presented to stakeholders. Using this approach, where you observe changes in the model as you alter parameters from the lowest to highest values you consider for your simulation, is critical.

We have performed sensibility tests on all individual components offered by CLEM ensuring that they work for the expected situation. While we have attempted to write generic code capable of handling the range of real world situations needed by users you may find something doesn't work as expected when applied to your simulation. These situations should be carefully examined and if believed to be a problem with the model, reported to APSIM Issues. We are confident that the components provided in CLEM work, and therefore the general flow of the simulation should be correct with complexity arising from the number and combination of these checked components used.

Any model code can, or probably will, contain bugs and logic errors. These issues may lay dormant, unnoticed, or not even affect the majority of simulation runs. The prevalence of bugs generally reduces with a combination of time since release and the number of people using the model and reporting these issues to be fixed.

Both APSIM Next Generation and CLEM should be considered relatively new models and you may encounter problems or strange behaviour in both model performance and the user interface, especially if you are using aspects not commonly used or in a way not previously tested. While APSIM and CLEM have processes to test code, bugs are likely to be present and any strange behaviour should be reported to APSIM Issues.

A log of issues reported and updates performed are available in the APSIM Git issues repository and in the What's New section of this help system.

As part of your healthy model scepticism we encourage you to build your simulation model from scratch and change parameters, check results, and confirm they meet your expectations as an iterative process. This will confirm that each expected process is being performed as expected. This approach will also really help interrogate your results to identify real world implications as it becomes difficult to consider all processes leading to a particular outcome.

Update: The 2025 major update for CLEM required considerable change to the core of the model and many activities and tools. Therefore the detailed checking of the initial release is outdated. Please report any issue you find with the new version or maintain your version at the release before the upgrade. This will result in not receiving any further APSIM updates and corrections but you will be using the stable release of CLEM.

While CLEM has been developed by a small team with one programmer (at time of writing), it is based on two precursor models with further small teams of scientific involvement using the findings of a broader research community, it also utilises the development and code repository process of the APSIM community including version control and automated testing. The code quality assurance and acceptance process and automated testing before release ensures clean code practices and working code, but it must be realised that those moderating code may not have the full detailed understanding of the detailed processes behind the different models.

This section relates to how transparent the model is in access to code, documentation, parameter reporting, all aimed at removing a "black box" style of model where it is difficult for the user to understand or have any trust in the model.

Example simulations can provide sensibility tests and allow a range of users to interoperate the same simulation.

CLEM provides a number of example simulations with the APSIM installation. These are designed to allow new users to explore simulation setup, functionality and outcomes. These examples also provide sensibility tests, especially the filtering example that provides an example of each filter and sort style with initial populations and reports to confirm the filtering was correct. CLEM will also include validation tests that will be performed before code updates to ensure functionality hasn't been broken.

More examples will be provided as developed and the user community may be able to share project simulations,

Good, up-to-date documentation can help overcome lack of transparency and make it clearer whether the level of abstraction is appropriate and what assumptions the model developer has made (Holzworth, 2021).

Documentation of both the underlying model process and instructions of using the model are critical.

This online reference manual provides a full description of each component offered by CLEM. These descriptions include all properties needed including the range of values permitted. We also describe the other components that add further functionality to the process.

The CLEM reference also provides overall explanations and walk-throughs to describe how to implement broader aspects such as cropping and grazing in CLEM.

A model is of little value if it can't easily be parameterised, run and results explored. We provide a full user guide along with the reference manual that describes how to build a simulation tree, produce output and use the various tools to explore the simulation performance.

We also offer links to videos presenting CLEM and have provided the responses we have given to a number of good questions asked by our users.

While there is some debate over whether clean code practices should include comments in the code base, CLEM does include comments where deemed necessary to help understand the code. The CLEM code is also documented by C# coding practices and includes descriptions of all classes, properties and methods including required arguments. These descriptions are provided in the intellisense tools of Visual Studio when developing code. CLEM code adheres to camelCase and PascalCase formatting and uses long descriptive parameter naming. The GrowPF code also include linkages to the Freer et al. 2012 equation numbers in parameter names.

One of the best approaches in interrogating a model performance is to look at output and the ability to access specific values is critical.

ASPIM Next Generation provides a report component that allows you to output any public property in the CLEM simulation. This does take some understanding of the system including accessing models and their properties and reporting on events. CLEM provides a number of reports that automatically handle this for you, and we also outline how to build your own custom reports. CLEM endeavourers to provide access to the vast majority of properties that you may wish to report with the intellisense helper in the Report proving lists of all properties available.

The model style implemented by CLEM allows significant customisation by the user allowing the model to represent a wide range of farm systems and management options. In allowing this approach much of the model functionality depends on the settings supplied by the user (even the arrangement of the Simulation tree). It is therefore critical that the user has the ability to ensure the model was doing what they expected.

One way of improving model transparency is the provision of the descriptive summary in CLEM where the entire simulation is provided in a plain English description to ensure you have set all the properties correctly and built the simulation tree as expected. This also provides a useful document to share and discuss with other researchers or stakeholders.

One of the biggest challenges in investigating the outcomes a complex whole-farm simulation is to understand when activities were undertaken and whether they were successfully performed. CLEM offers some visualisation tools to explore your model setup and the flow of the model through time. The Activities performed report displays each activity and whether it was performed.

A Labour requirements report also helps explore how the labour requirement components are applied in your simulation.

As resource shortfalls can have critical implications the Resource shortfalls report reports each time a resource was not available as required in the simulation.

We were also able to identify situations that we don't think you intended in the code. Many of these situations will report a warning to alert the user to what has happened. We do not make any decisions about whether the outcome was anticipated or not and so the model will run and we recommend you keep checking the ZoneCLEM component and the Messages tab for any warnings and information.

Full access to the model code base is important whether open source, or the source is made available by other means. Even when available messy code, poor variable naming conventions, or simply the complexity of code (e.g., number of files, classes, functions) gets in the way of transparency.

The full CLEM code base is open-source and available with APSIM Next generation. This code is managed in a Git code repository with full tracking of code changes through time, approved by moderators.

The open source nature of CLEM means anyone can commit changes and new functionality in future. Developing CLEM is an advanced task requiring significant understanding of the underlying process and code practices. We are developing an advanced section within the help system to assist with this task.

Positive past experiences with a model either by yourself or others will help build trust. You can access this experience by published papers using the model's user community.

As CLEM is a relatively new model (released Sept 2019) it only has a small number of users to date. There are a number of projects currently using CLEM both within and external to CSIRO. There is also a larger community of people who have used the two precursor models from which CLEM was derived (IAT and NABSA) with published results (see publications). Contact us to find out if others are using CLEM in your area of research or whether CLEM can perform your required simulations, as is, with minor improvements, or whether your project requires advanced skills to add new functionality to the model.

As you will often invest considerable effort and expense when starting to use a new model, it is an imperative to ensure the model will be maintained, supported and available into the future. While, no one can guarantee the future there are aspects you can consider to reduce future risk.

We believe CLEM is a very useful and novel model for simulating a wide variety of whole-farm systems and even other simulations utilising the resource-activity approach and believe uptake will continue into the future.

CSIRO has currently invested in the development of CLEM for use in a range of current and future research projects and our plan is to further our research collaboration with other organisations using CLEM. Any future development if the model is planned from research projects. CLEM is a key modelling tool in the livestock research at CSIRO and the ongoing investment in the development of new ruminant growth models shows ongoing commitment.

The fostering of an international community of users will also increase the skill and support base for CLEM. CLEMinar online sessions will be held when enough interests warrants them.

The APSIM Initiative is currently committed to supporting APSIM Next Generation for the range of Australian and international users.

The full APSIM code base has recently been reproduced in the latest programming language and is build in .Net Core for use on Windows, MacOS and Linux systems. This latest programming framework is up to date and easily maintained and compatible with current high performance computing. APSIM will be kept up to the latest long term support version of the .Net runtime.

APSIM code is open source and freely available to research and non-profit users (a commercial fee structure applies for commercial users). This means there is potentially an international team of users with access to maintain the code even if the original development team is available.

The open-source nature of the APSIM and CLEM code also means that new functionality added is immediately available to all users of CLEM in the next approved upgrade.

If you are adding new functionality to CLEM that is commercial in confidence and cannot be made freely available this will need to be undertaken in your development environment and may limit your ability to take advantage of future upgrades. Developing open source software in your research needs to be considered when organising IP and legal arrangements.

The APSIM code is being maintained with all reported issues addressed.

CLEM is being maintained and improved as used in projects. While the development team is working with CLEM we will be maintaining and supporting the model. It is hoped that a growing user community and some advanced users will also be able to support other users in future.

We are currently committed to helping early users of CLEM become competent users and build a strong community. We are able to help with various levels of support.

  • Freely discuss the suitability of CLEM for your research requirements.
  • Provide basic support via email to get simulations up and running. We expect you have read the online documentation and performed trouble shooting.
  • Provide funded workshops of hands on building of CLEM simulations.
  • Partake in collaborative research where new advanced functionality is required.

There can be considerable effort required to implement a full whole-farm simulation in CLEM. This relates to ensuring you have access to required data and parameters for your species/location. You will also require time to learn to use CLEM. A full understanding and familiarity is essential to fully trust both the result of the model and your building of the simulation. This will inform you as to whether the model is detailed enough for the questions you are asking.

Any whole-farm model is a simplified representation of a complex, real world system. In order to create the model a wide variety of processes are converted into code for a logical flow by the computer. Numerous decisions are made in how to represent and simplify these processes and thus assumptions are made, often dictated by the current user requirements at the time of building the model.

Many of the tools and activities (tasks) included in CLEM are relatively simple and do what they are expected to. For example, moving animals from one location to another will do just that, but there are other properties associated with the activity to provide further details. An example would be whether sucklings are moved with the mother, as the default implementation would not assume this if only moving the breeding females.

This situation also applies to terminology. While we have tried to use generic terms, there are different meanings and even units used by different disciplines or even groups. You must ensure the model is talking about the same thing you are. For example, there are numerous definitions of a Heifer, and you should know what CLEM assumes.

Other aspects of CLEM are more complicated such as the ruminant growth model and herd management. We have tried not to assume, or hard code, logic or values into the code and allow the user to turn all processes off or include activities to perform the task in another way. Regardless, there may be things assumed. For example, the herd growth model does not currently include any effects of temperature on animals.

While we have tried to identify and report assumptions, these can often be subconscious or biased toward our organisation's or nation's understanding or view of how things happen. A full understanding of the documentation should be a good start to understanding what CLEM assumes and give you confidence to ask the developers further questions.

CLEM will require external data and parameters (e.g. crop and pasture production data) from either field measures, other biophysical models, or even expert opinion. This may require access to other proprietary models or setup files. The parameterised components for specific species or breeds may be freely available, but if not, these may take considerable expert knowledge to define. Even when available, you should confirm any component is parameterised correctly for your simulation.

It may seem crazy, but often no one has ever really thought through the full scope of management undertaken on a farm. This needs to be considered from a process-based modelling perspective. This will usually be a collaborative approach between the model operator, the stakeholders, and experts. This is an essential aspect of any farm modelling project.

You need to determine whether CLEM can already perform your simulation out of the box when installed, or whether additional enhancements are required. These modifications may be relatively simple, or require the time of a CLEM expert developer to be included in your project costs.

 

See next section - Using CLEM