Not all addons need to be upgraded but it's always best to update the addons when updating EE to ensure compatibility.
In most cases they will still work but you may be a few messages in the dev log warning about the use of deprecated functions. These aren't critical but provide a warning that they may break in the future.
In a small number of cases you may find that some are more critical but addon developers are well ahead of the game in most cases and would most likely have already pushed through an update to fix the problem so that's why it's best to keep up with the updates.
If you find both EE and addon updates a bit laborious then I'd highly recommend Updater. It greatly speeds up the updating process for both.