Simulate, Test, Analyze: A Framework For Rigor


“Have Fun Storming The Castle!”

At the end of this fall semester, the second year students in the Academy rolled and carried their medieval mechanisms of mayhem to the SRHS track and we spent the afternoon watching the devices hurl lacrosse balls across the athletic field.This project was the final performance assessment of the semester and required that students design a gravitationally powered projectile launcher. This is an age old engineering/applied physics project.

Like many engineering projects done in high school, the physics principles governing the dynamics of the project are quite complicated, and ultimately the actual “application” of the science principles is often cursory. Students don’t have the background or mathematical abilities to to do the complex calculations needed to make an optimization adjustment to their mechanical device.This leads to the disconnection between the science content and engineering practice. Students don’t have the ability to make an informed decision about design choices. This is because it is difficult, very difficult.

Over the past few years I have been very interested in addressing this problem. This post discusses a framework that I have been working on to incorporate science into engineering projects. I think this framework allows high school students to engage in difficult scientific analysis without overwhelming them.

A Framework For Rigor

I won’t claim that this is a perfect solution, but so far I think we have experienced some success in creating a tighter relationship between science and engineering.  Last December I helped conduct a workshop at the NCCPA Professional Development Conference in Petaluma, CA. The name of the workshop was “NGSS, Prediction Reports and Your Science Class” and the point of this workshop was to give the attendees a framework for incorporating the Engineering standards into the science curriculum.  My co-presenter (Vipul Gupta) and I focused on the creation of prediction reports using computer simulations as a way to address two very important standards in the NGSS framework:

Using Simulations with Informed Input

Computer simulations are very popular in the educational space. They give teachers and students a virtual space where students can interact with virtual lab equipment or virtual objects that behave similarly to physical objects in the real world. With that said, they can fail to address students misconceptions because they do not always succeed in linking a conceptual model to the physical behavior. I also believe that the best simulations are ones that output data that can be analyzed with other scientific/mathematical tools. I also think that a good simulation requires that students provide meaningful input that gives them opportunities for analyzing the relationship between the input and the output.

Simulations used in engineering projects can be extremely helpful in addressing one of the main problems in engineering education. Students often design and build mechanical devices without understanding the physical principles that govern the design. The design process becomes an exercise in trial and error, or simply is reduced to copying a design from the internet.

To do a predictive analysis of a rocket’s flight, or a bridge’s structural performance is extremely difficult and often requires advanced mathematics and physics. Simulations can give the students the ability to analyze their designs and understand how changing the design inputs affects the output. Once again, it is important to find a simulation that requires students to understand the inputs and outputs.

Virtual Trebuchet

For example, in our project, students were introduced to an online Trebuchet simulation tool. This simulation tool is great because it requires that the student learn how to measure and calculate certain inputs. The students must have a working knowledge of rotational inertia, center of mass, and other concepts before they use the simulation. This was ideal for our project because it gave students a relevance and motivation . They had learn about these concepts in order to actually use the simulation. The students could then change certain inputs and see how that would change the efficiency of the design, or the range of the projectile. The point is that they needed physics knowledge in order to use the tool. They might not have the ability to know how the simulation eventually calculated the output, but they knew that the simulation required an understanding of the inputs.


Example Report

The Prediction Report

The next step is to ask the students to prepare a prediction report. This report is designed to get students to demonstrate their understanding of the inputs, display evidence of the required calculations or measurements needed to create the inputs and then analyze the simulation outputs. In the report for this project, I asked students to show a set of calculations and measurements for determining the center of mass of their throwing arm and the rotational inertia (moment of inertia). Students also had to provide similar information for the counterweight.  The students then had to run the simulation and document the outputs from the simulation.

The Test:  Data is Needed

The next step is to test the device. To make this step more rigorous and to be able to relate the scientific analytical process to the engineering process, it is crucial for the students to collect data that can be used to analyze the performance of their device/product and then reflect on how they would improve their design.

For this project, we decided to use high-speed video and Vernier’s LoggerPro video analysis software to plot the position of the projectile as it was launched from the device.

The Analysis

The analysis is actually broken into two parts. The first part requires a collection of calculations while the second part uses those calculations to make some qualitative assessments.

For example, in the above project, students had to use the collected position data from the video analysis tool to calculate the kinetic energy of the projectile and then the efficiency of the device. They had to be proficient at the analytical tool, which in itself requires physics content knowledge, providing once again an opportunity to apply scientific models in the analysis portion of this engineering project.

I have included the instructions for the analysis report here: Projectile Launcher Analysis Report.

Finally, students are given the opportunity to use the information gathered in the analysis report to reflect on their design, and more importantly use the information to inform how they would improve on a future design. I have included below the set of questions that I asked my students:

  1. Compare the efficiency calculation of the simulation to the efficiency rating that you calculated for your actual performance. Please describe why you think these values are not the same.
  2. Consider the design of your trigger. What design and fabrication decisions would you change in order to improve your trigger, AND explain WHY you would make those changes.
  3.  Consider the design of your sling. What design and fabrication decisions would you change in order to improve your sling, AND explain WHY you would make those changes.
  4. Consider the design of your release mechanism (called the nose). What design and fabrication decisions would you change in order to improve this mechanism, AND explain WHY you would make those changes.
  5. Consider the design of your arm. What design and fabrication decisions would you change in order to improve your arm, AND explain WHY you would make those changes.
  6. Consider the design of all other components and the overall design. What design and fabrication decisions would you change in order to improve your device (other than the trigger, sling and arm), AND explain WHY you would make those changes.


The overall design of this framework can be boiled down to this:

  • Engage students in a computer simulation that simplifies the process of modeling and analyzing a complex physical/chemical/biological process, but be sure that the simulation requires some conceptual and computational thinking.
  • When testing the performance of the design (bridge, rocket, etc.) make sure that the students are required to  collect data that can be analyzed and that once again demands that they apply their theoretical models.
  • Design an assessment that uses the analysis and gives the students an opportunity to make informed judgements of their designs for the purpose of redesign.

4 thoughts on “Simulate, Test, Analyze: A Framework For Rigor

  1. winstonwolff

    Hi Steve—

    Thank you for this insight into the use of computer simulations for teaching. Your point about making sure simulations require some conceptual and computational thinking makes perfect sense. But can you expand what that might mean in practice for a non-teacher? I suppose the amount of thinking required would need to vary depending on what level of student you are trying to teach, e.g. middle, high-school, or university? It sounds like the thinking required here is to know what terms like length, mass and inertia are and how to measure them so you can give them to the simulation. Are there other sorts of thinking that you wish could be exercised with simulations?

    Liked by 1 person

    1. Steve Temple Post author

      Great points Winston. Yes, I think that a student should be able to “probe” the simulation in order to assess its own validity. A simulation is written by humans, not nature and so is inherently leaving something out. It is an approximation. I think it would be good for students to be able to identify the strengths and limitations of the simulation. This would give them the ability to recognize that the simulation is not emulating nature, but rather simulating a model that is an approximation, or best effort to describe and model the physical universe. So I think it is important for the student to have the operational knowledge to be able to interact with the simulation, but I also think they should be able to critique its efficacy and be able to identify its limitations.


  2. Carol

    I love the idea that students have to actively compare their predictions and experimental outcomes in the end and discuss discrepancies. It would be interesting for me to know, what timeframes we are talking about for a) the teaching of the scientific concepts, b) the teaching of how to use the simulations plus generating a prediction report and c) the construction of the device plus testing.

    Liked by 1 person

    1. Steve Temple Post author

      Thanks Carol, usually the entire process takes about nine to twelve weeks. Our (actually borrowed, modified, etc.) process includes a research phase, a design phase, a fabrication phase, a testing phase, and then an analysis phase. One thing that we haven’t done very often is to then revisit the project to then allow students to improve their initial design to create a second version. I would love to do this, but time restrictions have made this difficult to implement.

      In order to give the students the skills and content knowledge to use the simulations, we use a parallel instruction model where the students are doing the engineering design work while they are engaged in actually learning about the scientific models that might help them with the design process. This is both good and bad. It adds relevance and context to the theoretical learning, but is often out of sync and this causes students to get a bit frustrated – “I wish we knew that a few weeks ago!”



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s