For my final blog post of the program, I’m going to share some success stories for our machine learning Breakout project. There are 4 seniors on this project, all of which have learned a ton over the last 2 months. Three of us had never even used Unity before, and we all now know how to create a basic game in Unity. So let’s talk about my individual successes.
Fairly early in the project, we were all working on our own to try and figure out Unity ML Agents, the black box program used to train neural network brains using reinforcement learning. We all found some good resources and shared them, but the biggest problem was getting versioning correct. We tried following the official documentation, but we were still running into issues. I eventually was able to find a comment on one of the tutorials that had suggested a specific set of versioning that ultimately worked. I shared this with the team.
While this was good, we were still all working independently on Unity tutorials and how to create a working paddle and ball. I saw this was going to cause issues when we started collaborating on code, and really pushed for using a python virtual environment for our ML Agents setup. Thus the following was born, and refined multiple times, in discord:
The above image are my set of instructions for setting up ML Agents for where we did all of our training in a separate GitHub repo. This gave us one common set of instructions to work from, and they worked.
The next big success story for me came when we were trying to figure out odd ball bouncing physics off of the paddle. Basically the agent would learn to get stuck against a wall, and the ball would continuously bounce vertically along the edge of that wall forever. While it did produce decent rewards, it did not meet the objective of the game which is to break all the bricks. This stumped our group for a long while.
Eventually I found a resource that talked about using Quaternions to handle the physics interaction. I had not known about these before but to simplify things they apply an angle of rotation on an object. So the solution was this, calculate how far the ball hit from the center of the paddle, and apply an angle of rotation for where the ball deflects off the paddle, which increases the further away the ball was from the paddle. This Quaternion solution finally solved the vertical ball bounce issue, because if the ball hit the edge of the paddle it should rotate and shoot away at a shallow 15 degree angle.
There were 2 other issues that still remained with physics. First, the ball would sometimes hit a brick in such a way that it would infinitely bounce horizontally across the level. This was actually a fairly simple solution, we ended up adding a very small amount of gravity (0.01 of normal). Eventually, the ball would start to fall down. It wasn’t enough to be noticeable, but just enough to fix the issue.
The final physics issue was around ball velocity. I still am not 100 percent sure what causes the issue, but I implemented a solution to this as well. Basically every time the ball hits the paddle, the velocity is reset to full. This is in addition to applying the Quaternion angle. All 3 of these physics issues were plaguing us for a few weeks, but I was able to figure out solutions and it felt great.
One final success story that was just in the last week. We were finally to the point to implement lives for the player and for the AI. The goal was every time a ball would go below the paddle, a life would be lost. The problem was every time the ball went below the paddle, all 5 lives were lost. One teammate Zach postulated the issue was we were deactivating the ball when it dropped down below the paddle, so it still existed and each update would lose another life. This made it look instantaneous for losing all 5 lives. He was right.
I was able to figure out a solution. In addition to deactivating the ball as it went down below the paddle, at the same time I transformed it to just above the top wall of the level, and set it’s velocity to 0. The outside of the wall has no physics, and with a tiny amount of gravity, the ball will ultimately just rest there. Bingo! Only 1 life lost when the ball goes below the paddle. When the ball is shot again using spacebar, it is first transformed back to inside the game area, reactivated, and then shot downward at an angle. Problem solved.
That last solution around lives felt really good to solve. It was a creative solution, and took a bit of time to debug, as it wasn’t working originally because the ball wouldn’t again shoot. That ended up meaning we just needed to set a boolean flag and then it worked fine. But when it did finally work, I felt like a real developer, and that is something sorely needed with the job market as it exists right now and getting no responses to job applications.
Happy coding and good luck out there fellow developers!
Leave a Reply