Every year’s recap post, I say the same thing: that it was a busy year. But man, I’ve really felt this one…
At the start of the year, I was still working for PickNik, although I was already looking for jobs and ended up leaving in March. During my time off, I went to the UK, Belgium, and Germany on a solo trip for almost the whole month of April. I decompressed by seeing new places, doing a healthy amount of walking and train travel, and meeting new and old friends and collaborators. Upon returning to Boston, I almost immediately presented at a conference and then started my new job at The AI Institute the following week. Between the start of my new job and preparing for ROSCon 2024 in Odense, Denmark this October, the middle of the year was unrelenting and went by shockingly fast.
It’s really after returning from ROSCon, in these last few months of the year, that I’ve been able to take a break. As a result, I bring you my fifth annual summary since starting this blog. Enjoy the read!
Open-Source and Community Contributions
Personal Projects
My first “popular” repository from 2021, TurtleBot3 Behavior Demos, didn’t really see much action until the very end of the year. I couldn’t easily upgrade it to the latest ROS version because it was based on Gazebo Classic, which is very close to end of life and not supported in the newest ROS 2 versions (Jazzy and later). As I’m not the biggest fan of configuring simulators, I was ready to just let the project die. However, I was randomly contacted by ElSayed ElSheikh a few weeks back and he did all the heavy lifting to adapt the repo to use the new version of Gazebo. Thanks to Sayed, this repo is now back in action and with some end-of-year promotion, made it across the 300 star mark!
PyRoboSim, on the other hand, received a lot of attention from me and from the community. In large part, this was because we used it for our ROSCon 2024 workshop (described in the next section). Independently of the workshop, this repo had:
- 2.0.0 release in May which upgraded to Ubuntu 24.04 / ROS 2 Jazzy and replaced the PyQt5 tool with PySide6 for more permissive licensing and better cross-platform support.
- 3.0.0 through 3.2.1 releases that related to major new features for the ROSCon 2024 workshop. This includes new actions to detect objects and open/close doors, simulated failures, delays, and battery usage, better ROS 2 interfaces, and lots of bug fixes.
- New contributors: My favorite thing about open source is getting people to contribute, and this year PyRoboSim had engagement from 3 students: Muhid Abid, Sanjeev Kumar, and Robin Müller. In fact, the next release (3.3.0) will likely be mostly from Sanjeev’s and Robin’s contributions. I also have to give a shout-out to my former colleague Erik Holum who assisted with some nice DevOps updates and testing random PRs when I wanted a second pair of eyes on bigger changes.
- Crossed 250 stars!
Finally, this year brought a new package to my portfolio — PyRoboPlan. This was released as a companion tool to my mega-post this year, How Do Robot Manipulators Move?. PyRoboPlan is my attempt at providing an easy-to-use, educational package to teach the basics of manipulator motion planning. This tool was very well-received, getting almost 250 stars in less than a year, already putting it on par with the other repositories I made several years ago.
Unfortunately, I haven’t worked on PyRoboPlan too much after its initial release. This is partially due to other commitments, but mostly because I’ve been doing similar work in C++ for my actual job, and it feels a little tedious to repeat the same thing on my spare time. However, if anyone is interested in contributing, there are open Git issues I’m happy to guide you on!
ROSCon 2024
In last year’s recap post, I also had a brief section about my first ever ROSCon experience at ROSCon 2023 in New Orleans. It was such a great time that I was determined to make it back, even if it required me to work on my spare time (spoiler: it did).
(If you want to learn more, check out ROSCon 2024: Highlights from Odense.)
As I mentioned at the start of my post, I took a solo trip to Europe in my time between jobs. Here is a testament to the power of networking (much to the dismay of all introverted engineers like myself): both of the collaborations that got me to ROSCon 2024 involved meeting people during this trip.
Just before I left PickNik at the start of year, I had already committed to work with my then colleague, Sebastian Jahr, on mentoring a Google Summer of Code (GSoC) project. The project we chose to sponsor was to integrate optimization-based motion planning from Drake into MoveIt 2.
One of the applicants to the project was Aditya Kamireddypalli from the University of Edinburgh. Not only did Aditya seem knowledgeable about optimization-based motion planning, but he came recommended by his labmate and former GSoC contributor, Peter David Fagan. I was in the UK for my trip, so I decided to make a stop in Edinburgh and meet Peter, Aditya, and the rest of their group for lunch and a tour of the robotics facilities on campus… despite Peter almost killing me earlier that day with the typical Scottish experience of running up Arthur’s Seat in wet and windy conditions.
About a week later, I took the train down to Germany where I first spent the weekend with Sebastian Jahr at his place in Heidelberg. I basically showed up and told him “we definitely need to work with Aditya”. And so we made it happen. While Sebastian and Aditya took the lead on this project, and I mostly provided support, they worked extra hard to get themselves a full talk at ROSCon 2024! The code is also available at https://github.com/moveit/moveit_drake.
Immediately after visiting Sebastian in Heidelberg, my next stop was an unplanned “midpoint” in Stuttgart, on my way to Munich to see some other friends. Here, I had the chance of meeting up with Christian Henkel at Bosch Research, who leads the ROS Deliberation Community Group that I participate in. Not only did I get the chance to practice a talk I had to give in a few weeks (it was the “excuse” for my visit), but I got to see some of the robotics work going on at Bosch.
Unrelated, I also had a great time in Stuttgart meeting strangers from the Internet at … an Irish pub? Anyways, Jan Eilers and Mathias Mantelli were super cool guys to hang out with. Check out their company, Sereact, as they’re working on some really impressive stuff in the logistics robotics space.
Back on topic… Christian was organizing a ROS Deliberation workshop for ROSCon 2024 and asked me to join him. Fast forwarding a few months, we gathered a crew of 6 people and delivered a full-day (8 hour) workshop on state machines, behavior trees, task planning, and model verification with ROS 2. The simulator that demonstrated all these technologies? None other than PyRoboSim. After the workshop, Christian continued to be gracious and asked me to help him co-lead the community group. So please join?
By the way, the workshop contents are available online at https://github.com/ros-wg-delib/roscon24-workshop.
Davide Faconti, David Oberacker, David Conner, Christian Henkel, Matthias Mayr, (honorary guest) Pontus Rosqvist, and… me!
Contributing to Existing Packages
In addition to my own packages, this year I’ve been helping out with a few other open-source tools that frankly have been in desperate need of maintenance.
MoveIt 2: While I don’t use MoveIt myself at the moment, I did get maintainer access during my time at PickNik, as open source was a big motivating factor for my joining in the first place. Over the year, I’ve been trying to help keep both the core repo and the tutorials in working shape. It’s been mostly bug fixes, updates, and releases, but it’s been satisfying helping people use this complicated tool and seeing what they can achieve.
A community highlight from this year is that some highly motivated engineers from Rivelin Robotics in Sheffield, UK made a bunch of quality contributions. Along with some of the other active MoveIt maintainers, we worked with Tom Noble and the rest of the Rivelin team to get some new MoveIt releases out the door for Humble, Jazzy, and Rolling. It had been a while since the last releases…
Rosbridge: This is another set of contributions that originated from PickNik work. Last year, I added ROS 2 action support to rosbridge_suite and roslibjs, and was given maintainer access after a spontaneous interaction at ROSCon 2023. This year, a newcomer from Poland, Błażej Sowa, appeared in a related repository (web_video_server) with some key fixes necessary for ROS 2 Jazzy and later. Błażej and I worked together this year to get official releases out the door for rosbridge_suite and web_video_server, and actually had the chance to meet over a “few” drinks at ROSCon 2024! Yet another reason why these conferences are so worthwhile.
PyTrees: I have been using PyTrees since 2020, but I’m back to using it for work at The AI Institute. For those unfamiliar, this tool is probably the most popular Python implementation of behavior trees, and I would consider it a close second in popularity and maturity behind BehaviorTree.CPP. Earlier this year, I had a chat with Daniel Stonier, the sole author and maintainer of PyTrees, and he was kind enough to grant me maintainer access! I haven’t done very much yet besides a few reviews and fixes, but I hope to help get releases out for the newer ROS distros, as well as contribute some of our infrastructure updates upstream when the time is right.
Lastly, I also made a simple tutorial for specifying custom gradients in Drake. This is the one repo in this section that did not need any maintenance help as it’s very active, but I didn’t know where else to fit this small anecdote.
Presentations
To round off this lengthy list of contributions, I was fortunate enough to be invited to a few presentations this year — in person and online.
- Managing Robot Autonomy with Behavior Trees at the Robotics Summit and Expo in Boston (May 2024). This was presented in collaboration with Andrew Stout at The AI Institute, who invited me to help him on his accepted talk. Unbeknownst to Andrew, I was in the process of interviewing to be his coworker, and ended up officially joining him the week after this talk. You can find our slides here.
- Robot Arm Motion Planning with PyRoboPlan at the Robotics Developer Day (July 2024). Ricardo Tellez from The Construct was kind enough to host me as one of the speakers for this great online event focused on robotics education with ROS. PyRoboPlan doesn’t use any ROS at the moment, but I’m not complaining about this opportunity!
- Practical Approaches to High-Level Behavior in Robotics at a Robotics Seminar in the University of New Hampshire (October 2024). This was a fun talk to prepare, which made me think hard about my own research philosophy — something I had never done before. I have to extend my thanks to Professor Momotaz Begum for hosting and showing me the work going on in her group, as well as my coworkers and Erik Holum for absolutely obliterating me with feedback on my practice talk. You can find the slides here.
- Appeared on Muhammad Luqman’s Podcast (November 2024). For those who don’t know him, Muhammad Luqman is an educator based in Pakistan that focuses on sharing practical robotics knowledge, primarily using ROS, on his online platform Robotisim. His podcast features a growing list of guests I consider to be good role models for the next generation of robotics engineers.
Actually, the reason that I was connected with Luqman was my post from earlier this year, Advice to Young Engineers. I wrote this to constructively rant about the declining state of engineering influencers, since platforms like LinkedIn and Twitter/X are mostly giving attention to hype-chasers and grifters who want to maximize their own engagement instead of providing valuable educational content. A major reason for most of my public content is to try combat this status quo. I may not be the smartest or most interesting person out there, but I do work hard and try to stay authentic. I want you to know that this unglamorous part of engineering is how you truly learn, and there are no shortcuts, recipes, or magic AI tools that can replace gaining experience the proper way. As such, I urge you to think critically about who you choose to follow as a role model, as it’s almost never the loudest voice in the room that gives you the most valuable information.
Back to the Research
Now we’re onto the “work” part of this post. The evident highlight is that I left PickNik in March to join The AI Institute in May. To make matters confusing, I had originally left Boston Dynamics to join PickNik in 2022. However, The AI Institute is led by the former founder of Boston Dynamics, Marc Raibert. Also, both organizations are primarily funded by Hyundai. So is this the same place or not? Does this count as “boomeranging” back to the same company? What am I even doing with my life?
Here’s the deal: Boston Dynamics focuses more on production whereas The AI Institute focuses on the research pillars of athletic intelligence, cognitive intelligence (both related to different software capabilities), organic hardware design, and ethics. This is generally the case, except for the part where Boston Dynamics still does research with Atlas and is collaborating with other institutes like Toyota Research Institute (TRI). I don’t have answers for you, but I do have facts. The point of The AI Institute is to focus on research, not on products; or if you prefer, the research output itself is the product.
Anyways, I had joined PickNik to work on open source, and it was becoming increasingly clear to me that being part of a startup mostly funded by customers and research grants means you foremost need to focus on bringing in business. I felt like the open-source core of the company was being thrown aside by putting more and more emphasis on product, and this led to frustration by many (myself included) at this abrupt cultural change. Ultimately, I didn’t see how to separate myself from the business side and stay satisfied with my day-to-day job without being unfair to the company, so I started looking for roles that were notably insulated from the business side. I found myself talking to The AI Institute down the road from my home, and was lucky to get into a research role here.
I have to say, though, that I am grateful for my experience at PickNik. Despite my complaints about the cultural shift, I undoubtedly did get to work on open source software. I learned a tremendous amount about C++, the ROS ecosystem, and motion planning from talented coworkers. The small company experience was also something I needed to “level up” my skills as a relatively new software engineer; it was after my experience at PickNik being thrown head first into big projects that I truly learned how to solve challenging software problems and lead projects, whereas in my previous roles I was mostly following instructions and trying to survive completing my own tasks. This was the main theme behind last year’s review, titled “Senior Engineer”.
As another bonus, it was this same scrappy small-company experience that had me working on projects with NASA Ames and Johnson research centers through the SBIR program. While the video below wasn’t something I led myself, I certainly played a part in getting MoveIt Pro in a state that allowed NASA Johnson to create this super cool demo.
So, I’ve now been back in a “Research Software Engineer” role for the last 8 months or so. I work on a team named Manipulation, Planning, and Learning (MaPLe), which is co-led by Professors Rob Platt and Stefanie Tellex. Our driving philosophy is to explore the use of skill learning and skill composition techniques to make robots perform difficult, contact-rich manipulation tasks. If that sounds overly broad, it’s because it is. We have about a dozen people working on different aspects of this problem. I personally have worked on three main tasks within this project so far.
- Skill composition: This is my primary role within the team. I think it was my experience applying task and motion planning (TAMP) and behavior abstractions in software (like behavior trees) to real robotics systems that got me the job. The fun twist in this project is that we’re exploring how these technologies work when the atomic skills are learned using approaches like imitation learning and reinforcement learning.
- Motion planning: We want to work with different types of manipulation platforms; single-arm vs. multi-arm, fixed vs. mobile base, legged vs. wheeled, and so on. Compared to many established motion planning frameworks, such as MoveIt, our work deals with more reactive, online planning to track the output of learned policies. We found that this requires new tools and solutions compared to the traditional “sense, plan, act” motion planning loop. Here, my recent experience as a MoveIt maintainer and on PyRoboPlan has proved invaluable, and we’ve thus far demonstrated an amazingly successful collaboration between our research and software engineering teams.
- Equivariant Diffusion Policy: Our team’s first flagship skill learning solution was Equivariant Diffusion Policy (EquiDiff), which was presented at the Conference on Robot Learning (CoRL) 2024, as an Outstanding Paper Award finalist no less! I spent a good chunk of time getting myself back in the groove of machine learning by taking the code for this paper and refactoring it to work within our repository. More importantly, I worked on updates to optimize for research agility; that is, the ability to quickly and easily make modifications to model architectures in the pursuit of new research directions. This was extremely challenging for me, but thankfully my colleagues were very supportive and kind as I asked them all sorts of basic questions about ML.
Source: https://equidiff.github.io/
I’d like to compare this experience so far with my first research software role at MIT, which you can learn more about in my 2020 and 2021 (part 1) reviews. My MIT role was my first official job as a software engineer, coming out of an almost 8-year career in tech support, training, and marketing for MathWorks. I was a newbie figuring out a whole new job function and landscape of open-source software tools and algorithms. I felt like I was learning alongside the graduate students in the group, and this density of learning was a perfect introduction to the robotics software engineering world.
After almost 5 years as a software developer, I feel like I’ve really aged (because I have). Unfortunately, it means I’m much more grumpy and opinionated about software — just ask any of my colleagues — but on a positive note I feel I have more to offer as a supporting character for cutting-edge research. I’m still not the main research ideas person, and I don’t aim to be because my teammates have much better ideas, but I pride myself in my ability to help make their ideas work well and unblock them as needed.
There is also an organizational difference between research in a small academic group compared to a research institute with something like 250 people at the time of writing. I’m less “alone” as a software engineer helping a bunch of students; there are more resources (both computational and people), which means we can all specialize a bit more and work on bigger, more collaborative projects. It does mean that research institutes have more process behind them since you need some structure to survive working on bigger teams. This is something that people who have not worked this way before struggle with as they feel it slows them down… because it actually does, in service of other benefits. So a big part of this role is to balance the rigor of professional software engineering with the need to move quickly and enable the creation of research output. Not only that, but many researchers here come directly from graduate school, where they are required to be “main characters” in producing and promoting their own first-author work. There are valid instances of bruised egos among researchers who feel like they have landed in a supporting role to someone else with similar qualifications, and this is tough to navigate unless there is a clearly communicated vision for a project team.
I would say this is easily the toughest part of the job, and the challenge is much more interpersonal than it is technical. If people are interested in this topic, I may consider writing a future blog post that dives into this in more detail. Don’t get me wrong, though: this is quite an interesting, multi-dimensional problem to be working on in addition to the research problems.
Summary
I feel like 8 months into my new job, in I’ve finally got used to the organization and project requirements. Now that our team has put down a workable layer of software infrastructure, I’m looking forward to shift gears and support my talented coworkers in helping flesh out some cool new research. Stay tuned for more updates as we begin to make robots do interesting manipulation tasks that combine skill learning and skill composition.
At the end of last year’s post, I said I wanted to learn about motion planning from the ground up, mentioning I was just starting to play with the Pinocchio library. I am, of course, glad to say that this goal was absolutely smashed in the creation of PyRoboPlan and the How do Robot Manipulators Move? blog post. Additionally, we’re using a lot of Pinocchio at my current job in our motion planning efforts, so the learning is continuing.
My main personal goal next year is to learn more about optimization-based planning and control. I did a little bit of work with Drake, but I’ve found it to be bit of a monolith. As an alternative, I’ve just barely started playing with Pinocchio’s CasADi interface, and there’s a lot there to build up to in implementing tools for trajectory optimization, model predictive control, and so on. This will be my only additional goal for next year, as maintaining everything else I already work takes enough time. Plus, I’m getting older and should focus more on keeping up my health and energy. It’s a marathon, not a sprint.
Most importantly, I’m looking forward to talking to more robotics people worldwide. I already have another possible invited talk within the US lined up in early 2025. Also, I had such an amazing time in ROSCon 2024 that I’ll be trying hard to have something for ROSCon 2025 in Singapore. Hopefully I’ll see many of you there?